From uwe_f_mayer at yahoo.com Tue May 1 01:05:55 2012 From: uwe_f_mayer at yahoo.com (Uwe F. Mayer) Date: Mon, 30 Apr 2012 16:05:55 -0700 (PDT) Subject: [pypy-dev] Patches for Pypy under cygwin Message-ID: <1335827155.24082.YahooMailNeo@web162906.mail.bf1.yahoo.com> I edited / hacked the sources and managed to compile PyPy under Cygwin based on Python 2.6.7 and gcc-4.5.3. If you find this of interest, the patches are available at http://www.tux.org/~mayer/cygwin/pypy/. In particular the work-around for the tm structure to get by without the tm_gmtoff and tm_zone fields may be of interest, as it allows compilation of the rctime module on systems that don't have it, such as some versions of SunOS. --Uwe -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue May 1 11:27:56 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 1 May 2012 11:27:56 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module Message-ID: Hi all, Following the blog post about STM, I would like to sollicitate your attention to come up with some better syntax for the 'transaction' module. * First question: 'transaction'. The name is kind of bogus, because it implies that it must be based on transactional memory. Such a name doesn't make sense if, say, you are running the single-core emulator version. What the module is about is to give a way to schedule tasks and run them in some unspecified order. * How about replacing the global functions 'transaction.add()' and 'transaction.run()' with a class, like 'transaction.Runner', that you need to instantiate and on which you call the methods add() and run(). If moreover the class has an '__exit__()' that redirects to run(), then you can use it like this: with transaction.Runner() as t: for block in blocks: t.add(do_stuff, block) And maybe, like concurrent.futures, a map() method --- although it seems to me like CPython was happily relegating the built-in map() in the corner of "advanced users only"; then adding a map() again seems a bit counter-productive --- it would look like that: with transaction.Runner() as t: t.map(do_stuff, blocks) * Note that the added transactions are only run on __exit__(), not when you call add() or map(). This might be another argument against map(). * The point of the examples above is that "t" can also be passed around in the transactions, and t.add() called on it again from there. Also, such a syntax nicely removes the need for any global state, and so it opens the door to nesting: you can write one of the above examples inside code that happens to run itself as a transaction from some unrelated outer Runner(). Right now, you cannot call transaction.run() from within a transaction --- and it doesn't make sense because we cannot tell if the transaction.add() that you just did were meant to schedule transactions for the outer or the future inner run(). That's what is better with this proposed new API. (Supporting it requires more work in the current implementation, though.) * Another issue. I think by now that we need special support to mean: "I want to end a transaction, then non-transactionally call this C function that will likely block for some time, and when it returns, I want to start the next transaction". This seem general enough to support various kinds of things, like calling select() or epoll_wait(). The question is what kind of support we want. I played with various ideas and I'll present the combination that satisfies me the most, but I'm open to any other suggestion. We could in theory support calling in-line the function, i.e. just call select() and it will break the current transaction in two. This is similar to the fact that select() in CPython releases and re-acquires the GIL. But it breaks the abstraction that every add() gives *one* transaction. It kind of goes against the general design of the API so far, which is that you add() things to do, but don't do them right now --- they will be done later. To voice it differently, I dislike this solution because you can break a working program just by adding a debugging "print" in the middle (assuming that "print" would also break the current transaction in two, like it releases the GIL in CPython). It would break the program because what used to be in the same transaction, no longer is: random things (done by other unrelated transactions) can suddenly have happened because you added a "print". The idea I'm playing with is two running modes: "breakable" vs "non-breakable". Say you have a "@breakable" decorator that you have to add explicitly on some of your Python functions. The transaction is breakable only when all functions in the call stack are @breakable. As soon as one non-breakable function is in the call stack, then the transaction is not breakable (to err on the side of safety). No clue if this would make any sense to the user, though. In the end a call to select() would either break the transaction in two (if the current mode is "breakable"), or, like now, in non-breakable mode it would turn the transaction inevitable (which is bad if the C call is blocking, because it blocks all other transactions too, but which is at least correct). Thanks for reading all my ranting. Ideas welcome... A bient?t, Armin. From lac at openend.se Tue May 1 12:03:22 2012 From: lac at openend.se (Laura Creighton) Date: Tue, 01 May 2012 12:03:22 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: Message from Armin Rigo of "Tue, 01 May 2012 11:27:56 +0200." References: Message-ID: <201205011003.q41A3M7f030349@theraft.openend.se> In a message of Tue, 01 May 2012 11:27:56 +0200, Armin Rigo writes: >* First question: 'transaction'. The name is kind of bogus, because >it implies that it must be based on transactional memory. Such a name >doesn't make sense if, say, you are running the single-core emulator >version. What the module is about is to give a way to schedule tasks >and run them in some unspecified order. The problem is that the csc namespace is overloaded with terms that already mean 'make-a-noun out of the verb I want to do', and the verb list that means 'do something' is a rather full namespace as well. Often you can get what you want by dropping the noun-ification -- so you would replace 'transaction' with 'transact'. This will not help the 'tranactional memory' association. So how about 'schedule'? My experience says that if you pick a verb, and not a noun, the code often comes out cleaner, because otherwise you tend to write -- or the people who use your library tend to write -- state-full things by habit even when they are not needed because the language points them that way. It looks to me as if your design is rather stateless by design, so maybe you want a verb and not a noun in any case. >* How about replacing the global functions 'transaction.add()' and >'transaction.run()' with a class, like 'transaction.Runner', that you >need to instantiate and on which you call the methods add() and run(). > If moreover the class has an '__exit__()' that redirects to run(), >then you can use it like this: > > with transaction.Runner() as t: > for block in blocks: > t.add(do_stuff, block) > >And maybe, like concurrent.futures, a map() method --- although it >seems to me like CPython was happily relegating the built-in map() in >the corner of "advanced users only"; then adding a map() again seems a >bit counter-productive --- it would look like that: > > with transaction.Runner() as t: > t.map(do_stuff, blocks) > >* Note that the added transactions are only run on __exit__(), not >when you call add() or map(). This might be another argument against >map(). Or an argument to build a different sort of 'add me lots of things' function?? I am not sure this makes sense. I am going kayaking overnight now, I will think of this more while paddling. >* The point of the examples above is that "t" can also be passed >around in the transactions, and t.add() called on it again from there. > Also, such a syntax nicely removes the need for any global state, and >so it opens the door to nesting: you can write one of the above >examples inside code that happens to run itself as a transaction from >some unrelated outer Runner(). Right now, you cannot call >transaction.run() from within a transaction --- and it doesn't make >sense because we cannot tell if the transaction.add() that you just >did were meant to schedule transactions for the outer or the future >inner run(). That's what is better with this proposed new API. >(Supporting it requires more work in the current implementation, >though.) But, for what it is worth, it pleases my aesthetic sense to no end. Which is why I want an 'add me lots of things' that works with this. :-) >* Another issue. I think by now that we need special support to mean: >"I want to end a transaction, then non-transactionally call this C >function that will likely block for some time, and when it returns, I >want to start the next transaction". This seem general enough to >support various kinds of things, like calling select() or >epoll_wait(). The question is what kind of support we want. > >I played with various ideas and I'll present the combination that >satisfies me the most, but I'm open to any other suggestion. > >We could in theory support calling in-line the function, i.e. just >call select() and it will break the current transaction in two. This >is similar to the fact that select() in CPython releases and >re-acquires the GIL. But it breaks the abstraction that every add() >gives *one* transaction. transaction == 'set of things to schedule, atomically'? or something else? Right now it is not clear to me why 'one' is important, probably because I do not understand something. >It kind of goes against the general design >of the API so far, which is that you add() things to do, but don't do >them right now --- they will be done later. To voice it differently, >I dislike this solution because you can break a working program just >by adding a debugging "print" in the middle (assuming that "print" >would also break the current transaction in two, like it releases the >GIL in CPython). It would break the program because what used to be >in the same transaction, no longer is: random things (done by other >unrelated transactions) can suddenly have happened because you added a >"print". I can agree that any program that breaks when you add a print is a bear to debig -- I used to get this quite often with Borland's C++ and it made me pull out hair. >The idea I'm playing with is two running modes: "breakable" vs >"non-breakable". Say you have a "@breakable" decorator that you have >to add explicitly on some of your Python functions. The transaction >is breakable only when all functions in the call stack are @breakable. > As soon as one non-breakable function is in the call stack, then the >transaction is not breakable (to err on the side of safety). No clue >if this would make any sense to the user, though. In the end a call >to select() would either break the transaction in two (if the current >mode is "breakable"), or, like now, in non-breakable mode it would >turn the transaction inevitable (which is bad if the C call is >blocking, because it blocks all other transactions too, but which is >at least correct). This seems up-side down to me. Do you want to decorate them as breakable? Or assume that they are breakable and decorate them as unbreakable? Maybe I am missing something crucial, but I think the other way may be easier for newly written code. Your way may be cooler for porting exisitng things, though, I need to think more. > >Thanks for reading all my ranting. Ideas welcome... > > >A bient??t > >Armin. Laura From arigo at tunes.org Tue May 1 12:55:36 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 1 May 2012 12:55:36 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: <201205011003.q41A3M7f030349@theraft.openend.se> References: <201205011003.q41A3M7f030349@theraft.openend.se> Message-ID: Hi Laura, On Tue, May 1, 2012 at 12:03, Laura Creighton wrote: > So how about 'schedule'? Might work :-) How about, if we decide to go for a class, the name of the class? Is it 'schedule.Runner()', 'schedule.Scheduler()', ...? A concrete instance of this class is no longer stateless, so 'schedule.Schedule()' is probably not so correct. >>* Note that the added transactions are only run on __exit__(), not >>when you call add() or map(). ?This might be another argument against >>map(). > > Or an argument to build a different sort of 'add me lots of things' > function?? ?I am not sure this makes sense. ?I am going kayaking > overnight now, I will think of this more while paddling. Yes, some kind of 'add lots of things' function. But I'm rather unsure it's worth creating a special function to do that. I'm happy with "for x in list: add(x)". > transaction == 'set of things to schedule, atomically'? or something > else? ?Right now it is not clear to me why 'one' is important, probably > because I do not understand something. One transaction == one unit of code that runs atomically. That's why I complained that by doing one add() you would end up with sometimes more than one transaction, if the function that you added breaks it in several ones by calling select() or other external functions. But maybe it's still correct if we go for the "breakable transaction" model. > This seems up-side down to me. ?Do you want to decorate them as > breakable? ?Or assume that they are breakable and decorate them as > unbreakable? No, I want it precisely the way I said it: by default, transactions must not be breakable. If the default was the other way around, it would mean that by default your program is not safe against random breakage --- pun intended: with breakable transactions your program risks to break. With non-breakable ones it is safe, but just potentially inefficient if it calls "print" or "select". So the @breakable decorator is something you would put only on, say, the function that does the loop on select(). Alternatively we could have two kinds of add(): one that adds regular and the other that adds breakable transactions. That would mean the transaction is then breakable *anywhere*, which is less safe. On the other hand, a @breakable decorator would be hard to implement in the emulator module to run on CPython... Argh. At least a "breakable-anywhere" transaction is closer to the notion of "spawn a new thread to run this in the background"... I have to think more, but it would seem that "breakable-anywhere" transactions would really offer a replacement for (simple usages of) threads. Maybe in the end it's the best solution: two kinds of transactions, and the "unsafe" one is the "breakable-anywhere" kind that really works like a thread, releasing the GIL around calls to blocking C syscalls --- except of course not with a real GIL in the real pypy-stm, but giving you this illusion. Bah, now I'm wondering if it could be used to reimplement the "thread" or "threading" module interface instead, at the user Python level. Maybe it can, and you have to use "thread.start_new_thread()" to start such breakable-anywhere transactions. It would give Yet Another bonus to pypy-stm: "merely" the fact that existing programs using threads are then able to use multiple cores too... A bient?t, Armin. From david at deadpansincerity.com Tue May 1 15:22:41 2012 From: david at deadpansincerity.com (David Miller) Date: Tue, 1 May 2012 14:22:41 +0100 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: <201205011003.q41A3M7f030349@theraft.openend.se> References: <201205011003.q41A3M7f030349@theraft.openend.se> Message-ID: On 1 May 2012 11:03, Laura Creighton wrote: > > So how about 'schedule'? > > My experience says that if you pick a verb, and not a noun, the code > often comes out cleaner Except that this seems particularly ambiguous - it's both a noun and a verb... End Bikeshedding. -- Love regards etc David Miller http://www.deadpansincerity.com 07854 880 883 -------------- next part -------------- An HTML attachment was scrubbed... URL: From info_2 at sify.com Tue May 1 14:47:04 2012 From: info_2 at sify.com (Edward Lee) Date: Tue, 1 May 2012 14:47:04 +0200 (CEST) Subject: [pypy-dev] HONG LEONG BANK (Malaysia) Message-ID: <20120501125139.D13B7232CB7@server17.seonhosting.de> Hello, I contacted you to assist in distributing the money left behind by my late client, Regards, Mr Edward Lee +60146308549 Fax:+60(0)321784290 From faassen at startifact.com Tue May 1 15:52:09 2012 From: faassen at startifact.com (Martijn Faassen) Date: Tue, 1 May 2012 15:52:09 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: Message-ID: Hey there, On Tue, May 1, 2012 at 11:27 AM, Armin Rigo wrote: > * First question: 'transaction'. ?The name is kind of bogus, because > it implies that it must be based on transactional memory. ?Such a name > doesn't make sense if, say, you are running the single-core emulator > version. ?What the module is about is to give a way to schedule tasks > and run them in some unspecified order. The module name also conflicts with the 'transaction' module that's already there to integrate database transactions (in particular the ZODB, but it's used for relational database integration too): http://pypi.python.org/pypi/transaction Regards, Martijn From arigo at tunes.org Tue May 1 15:55:58 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 1 May 2012 15:55:58 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> Message-ID: Re-hi, Now I did a full circle, and I'm wondering again if we couldn't go with the single primitive of an "atomic" object. You use it as "with atomic:", in a way quite traditional for Transactional Memory. Indeed, it seems after all possible to have the following model: a pypy-stm interpreter with the unmodified thread and threading modules, with just the addition of "atomic" (from some module, maybe "atomic", or "transaction" again, or "transact", or a better name). You can use it to ensure that a piece of code is run atomically; this would mean, in the sense of CPython's GIL, that the GIL is not released for the complete duration of the "with" statement. (On top of CPython, this cannot be fully achieved without hacking at the interpreter, but to a limited extent we can emulate it by saying at least the following: two threads that both want to run "with atomic" are serialized. Easy to do with just one lock.) On top of that, it is possible to write more nicely usable constructs in pure Python. For example, to write the existing transaction.add()/run(), we would write code that starts N threads, each polling a Queue and running the items in a "with atomic" block. Or we can write "schedule.Runner()" or similar. This approach has several advantages: the fact that the thread pooling logic is written in pure Python and tweakable; but also the fact that it is compatible with normal threads. This means all the doubts I had about blocking C calls are already resolved --- if you need really to do blocking C calls, do them in a separate thread. For example, in twisted, the loop calling select() would be written in a thread --- or likely in the main thread --- while a pool of extra threads runs the actual logic. (Not a normal thread pool, but one which uses "with atomic" to run each item.) People are already used to this approach --- with the exception that this gives the additional *huge* benefit of illusive serial execution, so locks&friends are useless. The new things here, when compared to CPython or PyPy-without-STM, are: (1) the ability to use multiple cores, by running transactions that go from one release-the-GIL to the next one (which is already existing as work-in-progress on CPython for Hardware TM); (2) the "with atomic" construct that I already proposed to CPython last year (and certainly is not new, but inspired directly from classical TM). Does it make any sense to you? A bient?t, Armin. From holger at merlinux.eu Tue May 1 16:48:55 2012 From: holger at merlinux.eu (holger krekel) Date: Tue, 1 May 2012 14:48:55 +0000 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> Message-ID: <20120501144855.GL10174@merlinux.eu> Hi Armin, On Tue, May 01, 2012 at 15:55 +0200, Armin Rigo wrote: > Re-hi, > > Now I did a full circle, and I'm wondering again if we couldn't go > with the single primitive of an "atomic" object. You use it as "with > atomic:", in a way quite traditional for Transactional Memory. i like this, more minimal, more obvious. > Indeed, it seems after all possible to have the following model: a > pypy-stm interpreter with the unmodified thread and threading modules, > with just the addition of "atomic" (from some module, maybe "atomic", > or "transaction" again, or "transact", or a better name). Maybe "atomic" could become a __pypy__ builtin and there could be a "ame" or so package which atomic-using apps could depend on? In any case, I really like the twist of "To remedy the GIL use AME" :) > You can use > it to ensure that a piece of code is run atomically; this would mean, > in the sense of CPython's GIL, that the GIL is not released for the > complete duration of the "with" statement. (On top of CPython, this > cannot be fully achieved without hacking at the interpreter, but to a > limited extent we can emulate it by saying at least the following: two > threads that both want to run "with atomic" are serialized. Easy to > do with just one lock.) > > On top of that, it is possible to write more nicely usable constructs > in pure Python. For example, to write the existing > transaction.add()/run(), we would write code that starts N threads, > each polling a Queue and running the items in a "with atomic" block. > Or we can write "schedule.Runner()" or similar. > > This approach has several advantages: the fact that the thread pooling > logic is written in pure Python and tweakable; but also the fact that > it is compatible with normal threads. This means all the doubts I had > about blocking C calls are already resolved --- if you need really to > do blocking C calls, do them in a separate thread. For example, in > twisted, the loop calling select() would be written in a thread --- or > likely in the main thread --- while a pool of extra threads runs the > actual logic. (Not a normal thread pool, but one which uses "with > atomic" to run each item.) People are already used to this approach > --- with the exception that this gives the additional *huge* benefit > of illusive serial execution, so locks&friends are useless. > > The new things here, when compared to CPython or PyPy-without-STM, are: > > (1) the ability to use multiple cores, by running transactions that go > from one release-the-GIL to the next one (which is already existing as > work-in-progress on CPython for Hardware TM); > > (2) the "with atomic" construct that I already proposed to CPython > last year (and certainly is not new, but inspired directly from > classical TM). > > Does it make any sense to you? To me so far yes. I am wondering how this all applies to the execnet-execution model, btw. (http://codespeak.net/execnet for those who wonder what i mean) remote_exec()s on the same gateway run currently in different threads and thus only send/receive needs to use "with atomic", right? best, holger From arigo at tunes.org Tue May 1 17:35:36 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 1 May 2012 17:35:36 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: <20120501144855.GL10174@merlinux.eu> References: <201205011003.q41A3M7f030349@theraft.openend.se> <20120501144855.GL10174@merlinux.eu> Message-ID: Hi Holger, On Tue, May 1, 2012 at 16:48, holger krekel wrote: > Maybe "atomic" could become a __pypy__ builtin and there could be a "ame" or > so package which atomic-using apps could depend on? In any case, > I really like the twist of "To remedy the GIL use AME" :) Yes, indeed, a private name in the __pypy__ module looks fine. The applications are supposed to use the "ame" module or package (or whatever name makes sense, but I'm getting convinced that "transaction" is not a good one). The "ame" module re-exports __pypy__._atomic as ame.atomic for general use, but also offers more stuff like the Runner class with add()/run() methods. Also, again, it doesn't necessarily make sense to force a lexically nested usage of ame.atomic, so we could replace __pypy__._atomic with two primitives __pypy__._atomic_start and _atomic_stop, re-exported in the "ame" module, and write the "ame.atomic" context manager in pure Python. > I am wondering how this all applies to the execnet-execution model, btw. > (http://codespeak.net/execnet for those who wonder what i mean) > remote_exec()s on the same gateway run currently in different threads > and thus only send/receive needs to use "with atomic", right? In my proposal, existing applications run fine, using multiple cores if they are based on multiple threads. You use "with atomic" to have an additional degree of synchronization when you don't want to worry about locks & friends (which should be *always*, but is still an optional benefit in this model). Maybe you're just talking about simplifying the implementation of execnet's channels to use "with atomic" instead of acquiring and releasing locks. Then yes, could be, as long as you remember that "with atomic" gives neither more nor less than its naive implementation: "don't release the GIL there". A bient?t, Armin. From skip at pobox.com Tue May 1 17:58:17 2012 From: skip at pobox.com (Skip Montanaro) Date: Tue, 1 May 2012 10:58:17 -0500 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> <20120501144855.GL10174@merlinux.eu> Message-ID: Armin, I apologize, I'm only half paying attention, trying to follow along. What does "ame" stand for? Is there some reason a more descriptive name isn't used? "Atomic MEmory"? "Armin's Magic Enterprise?" Thx, Skip On Tue, May 1, 2012 at 10:35 AM, Armin Rigo wrote: > Hi Holger, > > On Tue, May 1, 2012 at 16:48, holger krekel wrote: >> Maybe "atomic" could become a __pypy__ builtin and there could be a "ame" or >> so package which atomic-using apps could depend on? In any case, >> I really like the twist of "To remedy the GIL use AME" :) > > Yes, indeed, a private name in the __pypy__ module looks fine. ?The > applications are supposed to use the "ame" module or package (or > whatever name makes sense, but I'm getting convinced that > "transaction" is not a good one). ?The "ame" module re-exports > __pypy__._atomic as ame.atomic for general use, but also offers more > stuff like the Runner class with add()/run() methods. > > Also, again, it doesn't necessarily make sense to force a lexically > nested usage of ame.atomic, so we could replace __pypy__._atomic with > two primitives __pypy__._atomic_start and _atomic_stop, re-exported in > the "ame" module, and write the "ame.atomic" context manager in pure > Python. > >> I am wondering how this all applies to the execnet-execution model, btw. >> (http://codespeak.net/execnet for those who wonder what i mean) >> remote_exec()s on the same gateway run currently in different threads >> and thus only send/receive needs to use "with atomic", right? > > In my proposal, existing applications run fine, using multiple cores > if they are based on multiple threads. ?You use "with atomic" to have > an additional degree of synchronization when you don't want to worry > about locks & friends (which should be *always*, but is still an > optional benefit in this model). ?Maybe you're just talking about > simplifying the implementation of execnet's channels to use "with > atomic" instead of acquiring and releasing locks. ?Then yes, could be, > as long as you remember that "with atomic" gives neither more nor less > than its naive implementation: "don't release the GIL there". > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From amauryfa at gmail.com Tue May 1 18:17:58 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 1 May 2012 18:17:58 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> <20120501144855.GL10174@merlinux.eu> Message-ID: Hi, 2012/5/1 Skip Montanaro > I apologize, I'm only half paying attention, trying to follow along. > What does "ame" stand for? Is there some reason a more descriptive > name isn't used? "Atomic MEmory"? > "Armin's Magic Enterprise?" > Hey, you found the hidden meaning :) Actually it means "Automatic Mutual Exclusion", and it the official name of the feature: http://pypy.org/tmdonate.html -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From bokr at oz.net Wed May 2 05:51:48 2012 From: bokr at oz.net (Bengt Richter) Date: Tue, 01 May 2012 20:51:48 -0700 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> <20120501144855.GL10174@merlinux.eu> Message-ID: <4FA0AF54.9040007@oz.net> On 05/01/2012 08:35 AM Armin Rigo wrote: > Hi Holger, > > On Tue, May 1, 2012 at 16:48, holger krekel wrote: >> Maybe "atomic" could become a __pypy__ builtin and there could be a "ame" or >> so package which atomic-using apps could depend on? In any case, >> I really like the twist of "To remedy the GIL use AME" :) > > Yes, indeed, a private name in the __pypy__ module looks fine. The > applications are supposed to use the "ame" module or package (or > whatever name makes sense, but I'm getting convinced that > "transaction" is not a good one). The "ame" module re-exports > __pypy__._atomic as ame.atomic for general use, but also offers more > stuff like the Runner class with add()/run() methods. > > Also, again, it doesn't necessarily make sense to force a lexically > nested usage of ame.atomic, so we could replace __pypy__._atomic with > two primitives __pypy__._atomic_start and _atomic_stop, re-exported in > the "ame" module, and write the "ame.atomic" context manager in pure > Python. > >> I am wondering how this all applies to the execnet-execution model, btw. >> (http://codespeak.net/execnet for those who wonder what i mean) >> remote_exec()s on the same gateway run currently in different threads >> and thus only send/receive needs to use "with atomic", right? > > In my proposal, existing applications run fine, using multiple cores > if they are based on multiple threads. You use "with atomic" to have > an additional degree of synchronization when you don't want to worry > about locks& friends (which should be *always*, but is still an > optional benefit in this model). Maybe you're just talking about > simplifying the implementation of execnet's channels to use "with > atomic" instead of acquiring and releasing locks. Then yes, could be, > as long as you remember that "with atomic" gives neither more nor less > than its naive implementation: "don't release the GIL there". > I am looking at _____________________________________________________________________ def add(f, *args, **kwds): """Register the call 'f(*args, **kwds)' as running a new transaction. If we are currently running in a transaction too, the new transaction will only start after the end of the current transaction. Note that if the same or another transaction raises an exception in the meantime, all pending transactions are cancelled. """ r = random.random() assert r not in _pending # very bad luck if it is _pending[r] = (f, args, kwds) _____________________________________________________________________ from https://bitbucket.org/pypy/pypy/raw/stm-gc/lib_pypy/transaction.py and wondering about implementing atomicity guarantees in the evaluation of *args and **kwds, and maybe even f -- i.e., it would seem that there is opportunity effectively to pass arguments by value (once compiled), by reference, or by name, and arguments can be complex composites or simple constants. To prepare them they will be evaluated according to their expressions, either at call time or maybe partly at def time or construction time or method binding time or combinations. (When/how might multiple processors cooperate to evaluate arguments for passing into the transaction context? Never?) So how does one think about the state of arguments/values being accessed by f when it runs in its transaction context? I.e., if some arguments need to be version-synced, are there new ways to program that? How would you put the evaluation of function arguments into the inside of a transaction, e.g. if the arguments derive from stateful stuff that is updated as a side effect? Wrap it with an outer transaction? From the definition of transaction.run it appears that f must have its effect as a global side effect identified either through its arguments or built into its code (and presumably one could pass a bound method in place of f for an atomic update of instance attributes?). Seems like it might be nice to be able to pass a function and get back a list of function results? What should the common convention for accumulating results be with run as now? Should one pass f an additional queue argument to append to? Or an index into a selected slot in a global list, if ordering is predetermined. BTW,is side-by-side parallelism the only concern in the current attempt to run programs on many cores? What about pipelining-type parallelism, like nested generators with inner loops feeding outer ones running on different processors? I've played with the idea of generators in the form of classes that can be glued with '|' so they become logically one generator like (I've got a toy that will do this): for v in G(foo, seq)|G(bar)|G(baz): print v # pipe sequence-items through foo then bar then baz having the effect of for v in (baz(z) for z in (bar(r) for r in (foo(o) for o in seq))): print v or, I suppose, for o in seq: print baz(bar(foo(o))) but that's factored differently. Wondering how transaction stuff might play re transaction guarantees for generator states and serializing the feed from one to the next. Does it even make sense or does pipelining need a different kind of support? Re names: what about a mnemonic like pasifras == [p]arallel [as] [if] [ra]ndomly [s]erial Thence pasifras.add ? Regards, Bengt Richter From arigo at tunes.org Wed May 2 09:22:03 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 2 May 2012 09:22:03 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> Message-ID: Hi Laura, A note on my two previous messages: in the first one I argued about @breakable, while the second one I suggested to use threads and "with atomic". The relationship between the two is: in the first model you would put @breakable on some outer functions until it calls things that are supposed to be atomic; in the second model you would identify the call in question and put "with atomic" there. I would personally prefer @breakable because it forces the "atomic by default" idea a bit further, but I fear that the second model is better because it is fully compatible with existing programs. Moreover, as I argue repeatedly, regular apps should use neither, and only library implementors should have to worry. A bient?t, Armin. From arigo at tunes.org Wed May 2 09:43:34 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 2 May 2012 09:43:34 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> <20120501144855.GL10174@merlinux.eu> Message-ID: Hi Skip, On Tue, May 1, 2012 at 5:58 PM, Skip Montanaro wrote: > (ame) ?Is there some reason a more descriptive > name isn't used? That name was used as a temporary placeholder for a better name :-/ I think that obscure acronyms should be out (including "stm"). Laura's "schedule" was good, but does it still apply? Can you reasonably write "from schedule import atomic"? ... well, maybe, yes. So "schedule" would be the module in which we find at least "atomic", a pair of functions "atomic_start() / atomic_stop()", and the "Runner" class implementing a thread pool. Or maybe, keeping in touch with "concurrent.futures" from Python 3.2, should we make it "nonconcurrent.schedule"...? :-) A bient?t, Armin. From fijall at gmail.com Wed May 2 10:47:53 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 2 May 2012 10:47:53 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> <20120501144855.GL10174@merlinux.eu> Message-ID: A sidenote In CPython they introduced concurrent.* namespace. How about we reuse it, by say concurrent.{ame,transaction,stm}? Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed May 2 10:57:32 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 2 May 2012 10:57:32 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: References: <201205011003.q41A3M7f030349@theraft.openend.se> <20120501144855.GL10174@merlinux.eu> Message-ID: Hi Fijal, >> Or maybe, keeping in touch with "concurrent.futures" from Python 3.2, >> should we make it "nonconcurrent.schedule"...? :-) On Wed, May 2, 2012 at 10:47 AM, Maciej Fijalkowski wrote: > In CPython they introduced concurrent.* namespace. How about we reuse it, by > say concurrent.{ame,transaction,stm}? I'll take it that you didn't read the mail you are answering to...? As you can see with other mails on this very thread, it's quite hard to convince people that they need to think in terms of serial execution and not concurrent execution. (Sometimes, I even have to remind myself of this, after thinking in the bogus direction for a while.) I'm really, really unsure that putting the module in the "concurrent" package is going to help here... A bient?t, Armin. From lac at openend.se Thu May 3 12:38:16 2012 From: lac at openend.se (Laura Creighton) Date: Thu, 03 May 2012 12:38:16 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: Message from Armin Rigo of "Wed, 02 May 2012 09:22:03 +0200." References: <201205011003.q41A3M7f030349@theraft.openend.se> Message-ID: <201205031038.q43AcGOU014509@theraft.openend.se> In a message of Wed, 02 May 2012 09:22:03 +0200, Armin Rigo writes: >Hi Laura, > >A note on my two previous messages: in the first one I argued about >@breakable, while the second one I suggested to use threads and "with >atomic". The relationship between the two is: in the first model you >would put @breakable on some outer functions until it calls things >that are supposed to be atomic; in the second model you would identify >the call in question and put "with atomic" there. I would personally >prefer @breakable because it forces the "atomic by default" idea a bit >further, but I fear that the second model is better because it is >fully compatible with existing programs. Moreover, as I argue >repeatedly, regular apps should use neither, and only library >implementors should have to worry. > > >A bient??t, > >Armin. As promised, I have been doing more thinking while paddling. We camped out in G??teborg's archipelago and the island, as is common, has sheep. It occured to me that what we have is very like a shepherding problem. Since my uncle raised sheep, I have some personal experience with these problems from my childhood. 1. Assume you have 2 pastures, with a gate between them, and a flock of sheep. Your flock is in one pasture, and you want to get them to the other pasture through the gate. 2. Sheep are, to use the terms we have been using, 'unbreakable', and 'atomic'. (If you slice them into pieces to try to get them to fit through the gate, you will end up with non-functioning sheep on the other side). :-) 3. If your gate is narrow, you will have to serialise your sheep, and have them pass through one at a time. 4. If it is wider then parts of different sheep will pass through the gate intermingled from the perspective of a camera mounted on the gate. "Nose of sheep 1, nose of sheep 2, head of sheep 1, nose of sheep 3, head of sheep 3, body of sheep 3, tail of sheep 3, body of sheep 2, body of sheep 1, tail of sheep 2, tail of sheep 1" is what the camera might report as 'having gone passed', and we might conclude that sheep 3 is a small lamb, and that sheep 1 is its mother who slowed down going through the gate so that the lamb could keep up with her -- but all of this doesn't matter because, as long as you do not try to break them, the flock will function perfectly on the other side of the gate without any attention being paid to them by you. The main problem you have in moving pastures like this, as a shepherd, is that ewes (mother sheep) normally have twins or triplets (at least with the breed that my Uncle raised). Hungry lambs want to find their mothers right away, but the rest of the time they are busy frolicking with other lambs, butting heads, chasing each other, and doing other jeuvenile sheep things. They don't play with their siblings any more than with the other lambs. Ewes, on the other hand, want their own offspring close together, and with them all the time. The mechanism that the sheep use to keep track of each other is to bleat. Each lamb knows the sound of its mother's voice, and each ewe knows the sound of its own lambs'. They bleat more or less constantly when then are moving from a part of the pasture to another, and of course when they are being moved -- and they aren't all that quiet when grazing, either. If you are moving sheep through a gate, and it turns out that you have one lamb in the new pasture while it's sibling(s) are in the old pasture you can freeze your whole operation. The mother ewe will stand in your gate and refuse to move until she is reunited with all her lambs, which might be impossible if your gate is narrow. I don't think we have anything like this in our model, though maybe there is something analogous to bleating. Do we need to handle the case of bits of code that are themselves atomic that depend on being run with other bits of code in a predictable sequence, or one of several possible sequences? But you don't 'lock sheep', or sheep families. All you can do is try to submit them to the gate again. And from personal experience, it would be *very* *very* useful if you could magically restore the lamb that is early into the new pasture into the old pasture, right beside its mother. (Of course then you would just be able to teleport the whole lot and go away with gates altogether). At any rate, if we want to discourage people from using what they think they already know about concurrency and locking and whatnot, we might want to talk about what we are doing as herding a flock of . I'm pretty sure that this language isn't already being used somewhere to mean something else, at any rate. :-) You (or mostly your dogs) 'drive' and 'fetch' sheep through gates -- driving is away from the shepherd and fetching is towards the shepherd -- and you gather individual sheep into a flock. www.herding-dog-training-border-collie-sheepdog-dvd.com/herding_sheepdog_command_terminology.htm has some other interesting -- but I doubt useful words. But maybe, instead of scheduling, we are gathering? Not sure any of this helps at all, Laura From skip at pobox.com Thu May 3 12:57:12 2012 From: skip at pobox.com (Skip Montanaro) Date: Thu, 3 May 2012 05:57:12 -0500 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: <201205031038.q43AcGOU014509@theraft.openend.se> References: <201205011003.q41A3M7f030349@theraft.openend.se> <201205031038.q43AcGOU014509@theraft.openend.se> Message-ID: > But maybe, instead of scheduling, we are gathering? > > Not sure any of this helps at all, Laura, It might not help, but it was the most interesting Python-related post I've read in a long while. :-) Thanks, Skip From lac at openend.se Thu May 3 18:25:39 2012 From: lac at openend.se (Laura Creighton) Date: Thu, 03 May 2012 18:25:39 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: Message from Armin Rigo of "Wed, 02 May 2012 09:22:03 +0200." References: <201205011003.q41A3M7f030349@theraft.openend.se> Message-ID: <201205031625.q43GPdRB019901@theraft.openend.se> Listening to the whispers of your own creativity is always hard and fraught with error, but I am getting the strong sense that what we are up against is the idea that _processes_ are _things_ that are made up of _lines of code_ which _run_ on a _core_. Well, a sheep is a _thing_ in a very real sense that a flock is not 'a thing' but only 'a way of looking'. And the 'running' part can be thought of as 'happening all the time' -- it is just from time to time we need to provide a coherant way of looking at things -- to present them as if they were a story, with a beginning and end and which happened over this period of time, sequentially. So we are selecting what story to tell, leaving out many that could be told were we interested in a different one. This is an odd way to think. Sort of like how the Copenhagen explanation for quantum mechanics implies that classical concepts can be used to describe quantum phenomenon. You try, and try, and try to think that way until your brain hurts. And one day you explode, and say 'there are no things here', 'there are no concepts', THERE IS ONLY THE MATH AND THE MATH WORKS. At this point you can begin to get 10/10 rather than 0-3 on your problem sets again. I fear that using multiple cores may be a similar problem to those who are so very used to sequential operation. We need a non-sequential way to think, and so far we haven't been very good at this. Laura From bokr at oz.net Thu May 3 22:12:33 2012 From: bokr at oz.net (Bengt Richter) Date: Thu, 03 May 2012 13:12:33 -0700 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: <201205031625.q43GPdRB019901@theraft.openend.se> References: <201205011003.q41A3M7f030349@theraft.openend.se> <201205031625.q43GPdRB019901@theraft.openend.se> Message-ID: <4FA2E6B1.9070803@oz.net> On 05/03/2012 09:25 AM Laura Creighton wrote: > > Listening to the whispers of your own creativity is always hard and > fraught with error, but I am getting the strong sense that what we are > up against is the idea that _processes_ are _things_ that are made up > of _lines of code_ which _run_ on a _core_. > > Well, a sheep is a _thing_ in a very real sense that a flock is not 'a > thing' but only 'a way of looking'. And the 'running' part can be thought > of as 'happening all the time' -- it is just from time to time we need > to provide a coherant way of looking at things -- to present them as if > they were a story, with a beginning and end and which happened over this > period of time, sequentially. So we are selecting what story to tell, > leaving out many that could be told were we interested in a different > one. > > This is an odd way to think. Sort of like how the Copenhagen explanation > for quantum mechanics implies that classical concepts can be used to describe > quantum phenomenon. You try, and try, and try to think that way until your > brain hurts. And one day you explode, and say 'there are no things here', > 'there are no concepts', THERE IS ONLY THE MATH AND THE MATH WORKS. > > At this point you can begin to get 10/10 rather than 0-3 on your problem > sets again. > > I fear that using multiple cores may be a similar problem to those who > are so very used to sequential operation. We need a non-sequential way > to think, and so far we haven't been very good at this. > > Laura Hi Laura, Loved your pastoral imagery ;-) Apt metaphors are powerful, and fun to conjure, though difficult to get right. It struck me that there might be something to be learned from another example of dealing with sheep contending, and how solutions evolved. http://en.wikipedia.org/wiki/ALOHANET#The_ALOHA_protocol BTW, does the throughput graph imply anything for what you are doing? Regards, Bengt Richter From wlavrijsen at lbl.gov Fri May 4 08:16:28 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 3 May 2012 23:16:28 -0700 (PDT) Subject: [pypy-dev] cppyy: C++ bindings for PyPy Message-ID: Hi Maciej, On Mon, 23 Apr 2012, Maciej Fijalkowski wrote: > Quick question - if it's mature, why not merge it to default? I presume it > should be turned off, since there is a sizeable dependency, but still > having it in default can be good. I have a standalone version of Reflex now and made a few minor mods to cppyy to use it. That basically reduces the dependencies to Reflex (at runtime) and gccxml (for generation of reflection info, but not needed for the build or runtime). I'll update the documentation in a bit. What I'd like to have, is to defer the linking with libReflex until cppyy is imported into pypy-c. I've been unable to get that to work, though. :/ Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From matti.picus at gmail.com Fri May 4 11:15:22 2012 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 04 May 2012 12:15:22 +0300 Subject: [pypy-dev] win32-stdlib branch Message-ID: <4FA39E2A.2090602@gmail.com> I have modified files in lib-python/modified-2.7 to close all open files so that more of the lib-python tests pass in windows. The modifications live on the win32-stdlib branch, a recent diff to default shows: hg diff -r default --stat lib-python/modified-2.7/mailbox.py | 2179 +++++++++++++++++++++ lib-python/modified-2.7/tarfile.py | 60 +- lib-python/modified-2.7/test/test_mailbox.py | 1987 +++++++++++++++++++ lib-python/modified-2.7/test/test_old_mailbox.py | 160 + lib-python/modified-2.7/test/test_os.py | 825 +++++++ lib-python/modified-2.7/test/test_tarfile.py | 3 +- lib-python/modified-2.7/test/test_zipfile.py | 1372 +++++++++++++ lib-python/modified-2.7/zipfile.py | 1449 +++++++++++++ 8 files changed, 8013 insertions(+), 22 deletions(-) Who would like to review/reject/modify these changes? Matti From arigo at tunes.org Fri May 4 12:51:56 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 4 May 2012 12:51:56 +0200 Subject: [pypy-dev] Syntax for the 'transaction' module In-Reply-To: <201205031625.q43GPdRB019901@theraft.openend.se> References: <201205011003.q41A3M7f030349@theraft.openend.se> <201205031625.q43GPdRB019901@theraft.openend.se> Message-ID: Hi Laura, On Thu, May 3, 2012 at 6:25 PM, Laura Creighton wrote: > This is an odd way to think. ?Sort of like how the Copenhagen explanation > for quantum mechanics implies that classical concepts can be used to describe > quantum phenomenon. ?You try, and try, and try to think that way until your > brain hurts. ?And one day you explode, and say 'there are no things here', > 'there are no concepts', THERE IS ONLY THE MATH AND THE MATH WORKS. Right, in a way, as long as you remember that this applies only to the few people that need to dig under this particular cover. The Rule #1: we're writing a version of Python for general programmers, not for computer scientists Haskell fans. In fact almost everything about STM should be hidden even to the authors of the interpreter, just like GC is in PyPy. So I suppose that the quantum mechanics analogy works well if we take it this way: the quantum rules of the universe are strange and unexpected, but most people don't actually need them. You would need to understand them in order to grasp --- say --- how a laser *really* works, but not to build things on top of it, like a CD reader. For the latter, classical reasoning works fine. > I fear that using multiple cores may be a similar problem to those who > are so very used to sequential operation. ?We need a non-sequential way > to think, and so far we haven't been very good at this. If by "we" you mean "the general programmers" and not "the STM specialists", then I disagree :-) The STM model gives a way for the former people to continue to think in terms of sequential operations. This is true even if --- as an implementation technique only --- it's actually more complicated than that. But after all it has been more complicated than "sequential operations" for many years now, with out-of-order CPUs that give the illusion of sequential execution. And similarly when writing in Python we usually forget things like the GC, which gives automatic memory reclaim, or (from another point of view) the illusion of infinite storage space. It is all just an implementation technique, not something that the programmer has to tweak his brain around before taking the next step. A bient?t, Armin. From yselivanov.ml at gmail.com Fri May 4 18:46:16 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 4 May 2012 12:46:16 -0400 Subject: [pypy-dev] github pypy repo mirror Message-ID: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> Hello, It looks like https://github.com/pypy/pypy is out of sync. Can somebody take a look at that? Thanks! - Yury From Ronny.Pfannschmidt at gmx.de Fri May 4 19:03:11 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 04 May 2012 19:03:11 +0200 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> Message-ID: <4FA40BCF.3060003@gmx.de> Hi Yuri, That's not an official mirror, you'd have to get in touch with the one who made it -- Ronny On 05/04/2012 06:46 PM, Yury Selivanov wrote: > Hello, > > It looks like https://github.com/pypy/pypy is out of sync. Can somebody take a look at that? > > Thanks! > > - > Yury > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From blume.mike at gmail.com Fri May 4 20:54:35 2012 From: blume.mike at gmail.com (Michael Blume) Date: Fri, 4 May 2012 11:54:35 -0700 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: <4FA40BCF.3060003@gmx.de> References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> Message-ID: If people are likely to use it, I'd be happy to maintain a git mirror and transfer pull requests. -Mike On Fri, May 4, 2012 at 10:03 AM, Ronny Pfannschmidt wrote: > Hi Yuri, > > That's not an official mirror, > you'd have to get in touch with the one who made it > > -- Ronny > > > On 05/04/2012 06:46 PM, Yury Selivanov wrote: >> >> Hello, >> >> It looks like https://github.com/pypy/pypy is out of sync. ?Can somebody >> take a look at that? >> >> Thanks! >> >> - >> Yury >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From Ronny.Pfannschmidt at gmx.de Fri May 4 20:57:51 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 04 May 2012 20:57:51 +0200 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> Message-ID: <4FA426AF.2020106@gmx.de> personally i think unless you find a proper way to deal with the git has no named branches thing, this is a no-go -- Ronny On 05/04/2012 08:54 PM, Michael Blume wrote: > If people are likely to use it, I'd be happy to maintain a git mirror > and transfer pull requests. > > -Mike > > On Fri, May 4, 2012 at 10:03 AM, Ronny Pfannschmidt > wrote: >> Hi Yuri, >> >> That's not an official mirror, >> you'd have to get in touch with the one who made it >> >> -- Ronny >> >> >> On 05/04/2012 06:46 PM, Yury Selivanov wrote: >>> >>> Hello, >>> >>> It looks like https://github.com/pypy/pypy is out of sync. Can somebody >>> take a look at that? >>> >>> Thanks! >>> >>> - >>> Yury >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev From yselivanov.ml at gmail.com Fri May 4 21:05:38 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 4 May 2012 15:05:38 -0400 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: <4FA426AF.2020106@gmx.de> References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> Message-ID: <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> On 2012-05-04, at 2:57 PM, Ronny Pfannschmidt wrote: > personally i think unless you find a proper way to deal with the git has no named branches thing, this is a no-go Um, what is "named branches" thing that git apparently doesn't have? - Yury From Ronny.Pfannschmidt at gmx.de Fri May 4 21:06:57 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 04 May 2012 21:06:57 +0200 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> Message-ID: <4FA428D1.5000000@gmx.de> On 05/04/2012 09:05 PM, Yury Selivanov wrote: > On 2012-05-04, at 2:57 PM, Ronny Pfannschmidt wrote: > >> personally i think unless you find a proper way to deal with the git has no named branches thing, this is a no-go > > Um, what is "named branches" thing that git apparently doesn't have? named branches are branches that are part of the history git branches are only pointers, not part of the history From yselivanov.ml at gmail.com Fri May 4 23:00:46 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 4 May 2012 17:00:46 -0400 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: <4FA428D1.5000000@gmx.de> References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> <4FA428D1.5000000@gmx.de> Message-ID: On 2012-05-04, at 3:06 PM, Ronny Pfannschmidt wrote: > On 05/04/2012 09:05 PM, Yury Selivanov wrote: >> On 2012-05-04, at 2:57 PM, Ronny Pfannschmidt wrote: >> >>> personally i think unless you find a proper way to deal with the git has no named branches thing, this is a no-go >> >> Um, what is "named branches" thing that git apparently doesn't have? > > named branches are branches that are part of the history > git branches are only pointers, not part of the history OK. And why is it a 'no-go' for a mirror (not a repository that core pypy devs working with)? - Yury From blume.mike at gmail.com Sat May 5 03:09:45 2012 From: blume.mike at gmail.com (Michael Blume) Date: Fri, 4 May 2012 18:09:45 -0700 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> <4FA428D1.5000000@gmx.de> Message-ID: Yury: No idea -- seems like it should work just fine to me =). On Fri, May 4, 2012 at 2:00 PM, Yury Selivanov wrote: > On 2012-05-04, at 3:06 PM, Ronny Pfannschmidt wrote: > >> On 05/04/2012 09:05 PM, Yury Selivanov wrote: >>> On 2012-05-04, at 2:57 PM, Ronny Pfannschmidt wrote: >>> >>>> personally i think unless you find a proper way to deal with the git has no named branches thing, this is a no-go >>> >>> Um, what is "named branches" thing that git apparently doesn't have? >> >> named branches are branches that are part of the history >> git branches are only pointers, not part of the history > > OK. ?And why is it a 'no-go' for a mirror (not a repository that core pypy devs working with)? > > - > Yury > From Ronny.Pfannschmidt at gmx.de Sat May 5 08:55:53 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sat, 05 May 2012 08:55:53 +0200 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> <4FA428D1.5000000@gmx.de> Message-ID: <4FA4CEF9.2080406@gmx.de> its not a no go for a mirror, *I* consider it a no-go for the 2-way sync Michael suggested On 05/05/2012 03:09 AM, Michael Blume wrote: > Yury: No idea -- seems like it should work just fine to me =). > > On Fri, May 4, 2012 at 2:00 PM, Yury Selivanov wrote: >> On 2012-05-04, at 3:06 PM, Ronny Pfannschmidt wrote: >> >>> On 05/04/2012 09:05 PM, Yury Selivanov wrote: >>>> On 2012-05-04, at 2:57 PM, Ronny Pfannschmidt wrote: >>>> >>>>> personally i think unless you find a proper way to deal with the git has no named branches thing, this is a no-go >>>> >>>> Um, what is "named branches" thing that git apparently doesn't have? >>> >>> named branches are branches that are part of the history >>> git branches are only pointers, not part of the history >> >> OK. And why is it a 'no-go' for a mirror (not a repository that core pypy devs working with)? >> >> - >> Yury >> From sdouche at gmail.com Sat May 5 09:44:28 2012 From: sdouche at gmail.com (Sebastien Douche) Date: Sat, 5 May 2012 09:44:28 +0200 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: <4FA428D1.5000000@gmx.de> References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> <4FA428D1.5000000@gmx.de> Message-ID: On Fri, May 4, 2012 at 9:06 PM, Ronny Pfannschmidt wrote: >> Um, what is "named branches" thing that git apparently doesn't have? > > named branches are branches that are part of the history > git branches are only pointers, not part of the history Uh?! Non sense. Both backend have the full DAG. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche From blume.mike at gmail.com Sat May 5 09:59:50 2012 From: blume.mike at gmail.com (Michael Blume) Date: Sat, 5 May 2012 00:59:50 -0700 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> <4FA428D1.5000000@gmx.de> Message-ID: Sebastien: See my remark to Yury. Mercurial *actively* annotates your commits with information about what branch you were on when you made the commit. Git users can, of course, add this information, they just have to do it manually. Ronny: As I said, it's *really easy* to construct git commits which, when translated, come out on a named branch. On Sat, May 5, 2012 at 12:44 AM, Sebastien Douche wrote: > On Fri, May 4, 2012 at 9:06 PM, Ronny Pfannschmidt > wrote: >>> Um, what is "named branches" thing that git apparently doesn't have? >> >> named branches are branches that are part of the history >> git branches are only pointers, not part of the history > > Uh?! Non sense. Both backend have the full DAG. > > -- > Sebastien Douche > Twitter: @sdouche / G+: +sdouche > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From amauryfa at gmail.com Sat May 5 18:53:54 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sat, 5 May 2012 18:53:54 +0200 Subject: [pypy-dev] win32-stdlib branch In-Reply-To: <4FA39E2A.2090602@gmail.com> References: <4FA39E2A.2090602@gmail.com> Message-ID: 2012/5/4 Matti Picus > I have modified files in lib-python/modified-2.7 to close all open files > so that more of the lib-python tests pass in windows. CPython 3 emits a warning when a file is closed by its __del__ method. And of course the test suite has been updated to correctly close all files during the tests. Have you compared your changes with the ones in the py3k branch? -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From xancorreu at gmail.com Sat May 5 21:01:19 2012 From: xancorreu at gmail.com (xancorreu) Date: Sat, 05 May 2012 21:01:19 +0200 Subject: [pypy-dev] alioth tests? Message-ID: <4FA578FF.6060304@gmail.com> Hi, I'm just curious: can you make the tests of alioth [http://shootout.alioth.debian.org/u32q/which-programming-languages-are-fastest.php]? If you do, you can compare pypy to cpython and also to other programming languages. What do you think about this idea? Thanks in advance, Xan. From sebastien.volle at gmail.com Sat May 5 21:28:55 2012 From: sebastien.volle at gmail.com (=?ISO-8859-1?Q?S=E9bastien_Volle?=) Date: Sat, 5 May 2012 21:28:55 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: <4FA578FF.6060304@gmail.com> References: <4FA578FF.6060304@gmail.com> Message-ID: Hello, If I'm not mistaken, the policy of the language benchmark game is to have only one implementation of a specific language, and preferably the reference implementation. So I guess that unless pypy eventually becomes the reference python interpreter, CPython will remain the only python interpreter to appear in that benchmark. -- S?bastien 2012/5/5 xancorreu > Hi, > > I'm just curious: can you make the tests of alioth [ > http://shootout.alioth.**debian.org/u32q/which-** > programming-languages-are-**fastest.php > ]? > If you do, you can compare pypy to cpython and also to other programming > languages. What do you think about this idea? > > > Thanks in advance, > Xan. > ______________________________**_________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/**mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Sat May 5 21:49:48 2012 From: matti.picus at gmail.com (Matti Picus) Date: Sat, 05 May 2012 22:49:48 +0300 Subject: [pypy-dev] win32-stdlib branch In-Reply-To: References: <4FA39E2A.2090602@gmail.com> Message-ID: <4FA5845C.8010109@gmail.com> On 5/05/2012 7:53 PM, Amaury Forgeot d'Arc wrote: > 2012/5/4 Matti Picus > > > I have modified files in lib-python/modified-2.7 to close all open > files so that more of the lib-python tests pass in windows. > > > CPython 3 emits a warning when a file is closed by its __del__ method. > And of course the test suite has been updated to correctly close all > files during the tests. > Have you compared your changes with the ones in the py3k branch? > > -- > Amaury Forgeot d'Arc Ahh. No, I didn't. I'm not sure how to proceed. Perhaps it makes more sense to wait for the stdlib-unification branch to be merged, and then somehow compare the win32-stdlb branch's stdlib 2.7 to the py3k branch's stdlib 3.2 . Matti From xancorreu at gmail.com Sat May 5 21:52:14 2012 From: xancorreu at gmail.com (xancorreu) Date: Sat, 05 May 2012 21:52:14 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: References: <4FA578FF.6060304@gmail.com> Message-ID: <4FA584EE.6080902@gmail.com> No, I said to compare to the same tests (source files) of alioth. I explain: I see in http://speed.pypy.org/comparison/ that you run "ai", "bm_chameleon", .... tests. I suppose this was python sources. I say to use the same scripts of alioth (in addition of yours), because this, you can compare to cpython and additionally you (and users) could have if the pypy faster than cpython and also faster and language X. Maybe for you is not "important" but for example I'm amazing if I see that pypy is "only" 1.5x slower than C++ in alioth tests. Regards, Xan. Al 05/05/12 21:28, En/na S?bastien Volle ha escrit: > Hello, > > If I'm not mistaken, the policy of the language benchmark game is to > have only one implementation of a specific language, and preferably > the reference implementation. > So I guess that unless pypy eventually becomes the reference python > interpreter, CPython will remain the only python interpreter to appear > in that benchmark. > > -- > S?bastien > > 2012/5/5 xancorreu > > > Hi, > > I'm just curious: can you make the tests of alioth > [http://shootout.alioth.debian.org/u32q/which-programming-languages-are-fastest.php]? > If you do, you can compare pypy to cpython and also to other > programming languages. What do you think about this idea? > > > Thanks in advance, > Xan. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ronny.Pfannschmidt at gmx.de Sat May 5 22:12:54 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sat, 05 May 2012 22:12:54 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: <4FA584EE.6080902@gmail.com> References: <4FA578FF.6060304@gmail.com> <4FA584EE.6080902@gmail.com> Message-ID: <4FA589C6.803@gmx.de> That got already covered some months ago, my basic understanding of the incidents that happened back then is, that alioth is not a desirable test to participate in, the code/scripts are biased towards particular implementations, and the maintainer has some strange ideas, as far as i remember pypy was part of it for some time, but actually underperformed since scripts where heavily cpython-optimized which does not always help on pypy -- Ronny On 05/05/2012 09:52 PM, xancorreu wrote: > No, I said to compare to the same tests (source files) of alioth. I > explain: I see in http://speed.pypy.org/comparison/ that you run "ai", > "bm_chameleon", .... tests. I suppose this was python sources. I say to > use the same scripts of alioth (in addition of yours), because this, you > can compare to cpython and additionally you (and users) could have if > the pypy faster than cpython and also faster and language X. Maybe for > you is not "important" but for example I'm amazing if I see that pypy is > "only" 1.5x slower than C++ in alioth tests. > > Regards, > Xan. > > Al 05/05/12 21:28, En/na S?bastien Volle ha escrit: >> Hello, >> >> If I'm not mistaken, the policy of the language benchmark game is to >> have only one implementation of a specific language, and preferably >> the reference implementation. >> So I guess that unless pypy eventually becomes the reference python >> interpreter, CPython will remain the only python interpreter to appear >> in that benchmark. >> >> -- >> S?bastien >> >> 2012/5/5 xancorreu > >> >> Hi, >> >> I'm just curious: can you make the tests of alioth >> [http://shootout.alioth.debian.org/u32q/which-programming-languages-are-fastest.php]? >> If you do, you can compare pypy to cpython and also to other >> programming languages. What do you think about this idea? >> >> >> Thanks in advance, >> Xan. >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From blume.mike at gmail.com Sat May 5 23:04:31 2012 From: blume.mike at gmail.com (Michael Blume) Date: Sat, 5 May 2012 14:04:31 -0700 Subject: [pypy-dev] github pypy repo mirror In-Reply-To: References: <0C5BF014-7489-48B0-8942-B387351749E5@gmail.com> <4FA40BCF.3060003@gmx.de> <4FA426AF.2020106@gmx.de> <2E2BC1B0-3916-45E9-B095-E7FB2D4D28F7@gmail.com> <4FA428D1.5000000@gmx.de> Message-ID: http://www.github.com/MichaelBlume/pypy Got it on a 10 minute cron Note that this is *not* hash-compatible with the previous git mirror -- something changed in the hg-git implementation since that was started. On May 5, 2012 12:59 AM, "Michael Blume" wrote: > Sebastien: See my remark to Yury. Mercurial *actively* annotates your > commits with information about what branch you were on when you made > the commit. Git users can, of course, add this information, they just > have to do it manually. > > Ronny: As I said, it's *really easy* to construct git commits which, > when translated, come out on a named branch. > > On Sat, May 5, 2012 at 12:44 AM, Sebastien Douche > wrote: > > On Fri, May 4, 2012 at 9:06 PM, Ronny Pfannschmidt > > wrote: > >>> Um, what is "named branches" thing that git apparently doesn't have? > >> > >> named branches are branches that are part of the history > >> git branches are only pointers, not part of the history > > > > Uh?! Non sense. Both backend have the full DAG. > > > > -- > > Sebastien Douche > > Twitter: @sdouche / G+: +sdouche > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sun May 6 21:06:02 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 6 May 2012 13:06:02 -0600 Subject: [pypy-dev] feedback on PEP 421 Message-ID: Hi, I'm looking for feedback on PEP 421: "Adding sys.implementation". The idea came up in 2009 and garnered positive feedback, but didn't go anywhere. I've revived it and am hoping to get it worked out in time for Python 3.3. Any feedback would be very helpful, particularly with regard to the decision on the type of sys.implementation and the constraints on sys.implementation.version. Thanks. -eric http://www.python.org/dev/peps/pep-0421/ From arigo at tunes.org Sun May 6 21:29:12 2012 From: arigo at tunes.org (Armin Rigo) Date: Sun, 6 May 2012 21:29:12 +0200 Subject: [pypy-dev] feedback on PEP 421 In-Reply-To: References: Message-ID: Hi Eric, On Sun, May 6, 2012 at 9:06 PM, Eric Snow wrote: > I'm looking for feedback on PEP 421: "Adding sys.implementation". I will leave others to comment on that. For myself, I can only say that it looks like a good idea, which we will happily adhere to when we migrate to Python 3.3. Maybe others have more precise ideas than I do, but I'm fine with whatever details. The reason for that: getting details "right" might be similar to the sys module's documentation, which has grown a number of "CPython implementation detail" boxes --- which are rather meaningless for PyPy, in the sense that the set of functions that are dubbed "CPython implementation detail" and the set of functions that PyPy cannot reasonably implement have little to do with each other. So I'm only vaguely +1 on "sys.implementation", because what is actually an "implementation detail" and what is not might not be completely clear, even considering the four different implementations so far. I like the part of the PEP that tries to limit what is in there, instead of adding tons of stuff. A bient?t, Armin. From lac at openend.se Sun May 6 22:29:37 2012 From: lac at openend.se (Laura Creighton) Date: Sun, 06 May 2012 22:29:37 +0200 Subject: [pypy-dev] feedback on PEP 421 In-Reply-To: Message from Armin Rigo of "Sun, 06 May 2012 21:29:12 +0200." References: Message-ID: <201205062029.q46KTbCl018261@theraft.openend.se> In a message of Sun, 06 May 2012 21:29:12 +0200, Armin Rigo writes: >The reason for that: getting details "right" might be similar to the >sys module's documentation, which has grown a number of "CPython >implementation detail" boxes --- which are rather meaningless for >PyPy, in the sense that the set of functions that are dubbed "CPython >implementation detail" and the set of functions that PyPy cannot >reasonably implement have little to do with each other. I think a large part of this problem is that CPython implementors are completely unaware of what set of functions PyPy cannot implement. And I don't blame them for that. I am often in the same way. I talk to somebody who wants to make CPython do X, and he wants to know if PyPy will have a problem with this. I cautiously reply 'not as he has outlined the problem'. 3 weeks later, he comes back with working code. And I say -- Eeek! we cannot do this, that and the other thing, and by the way we have a much more efficient way to do this thing over there. He quite reasonably asks me 'why didn't you tell me this in the first place?' and I also reasonably reply 'I had no idea that you planned this particular implementation when you asked.' Is there a place in this PEP for 'known differences in implementations' a.k.a. 'we know this is incompatible but we aren't going to change our implementation to fix'?? It might save a lot of trouble. Laura From xancorreu at gmail.com Sun May 6 23:12:19 2012 From: xancorreu at gmail.com (xancorreu) Date: Sun, 06 May 2012 23:12:19 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: <4FA589C6.803@gmx.de> References: <4FA578FF.6060304@gmail.com> <4FA584EE.6080902@gmail.com> <4FA589C6.803@gmx.de> Message-ID: <4FA6E933.7050103@gmail.com> Al 05/05/12 22:12, En/na Ronny Pfannschmidt ha escrit: > That got already covered some months ago, > > my basic understanding of the incidents that happened back then is, > that alioth is not a desirable test to participate in, > the code/scripts are biased towards particular implementations, > and the maintainer has some strange ideas, What "strange ideas" mean? By the other hand, saying with other words: with scripts appeared in alioth, pypy is faster? Thanks, Xan. > as far as i remember pypy was part of it for some time, but actually > underperformed since scripts where heavily cpython-optimized > which does not always help on pypy > > -- Ronny > > On 05/05/2012 09:52 PM, xancorreu wrote: >> No, I said to compare to the same tests (source files) of alioth. I >> explain: I see in http://speed.pypy.org/comparison/ that you run "ai", >> "bm_chameleon", .... tests. I suppose this was python sources. I say to >> use the same scripts of alioth (in addition of yours), because this, you >> can compare to cpython and additionally you (and users) could have if >> the pypy faster than cpython and also faster and language X. Maybe for >> you is not "important" but for example I'm amazing if I see that pypy is >> "only" 1.5x slower than C++ in alioth tests. >> >> Regards, >> Xan. >> >> Al 05/05/12 21:28, En/na S?bastien Volle ha escrit: >>> Hello, >>> >>> If I'm not mistaken, the policy of the language benchmark game is to >>> have only one implementation of a specific language, and preferably >>> the reference implementation. >>> So I guess that unless pypy eventually becomes the reference python >>> interpreter, CPython will remain the only python interpreter to appear >>> in that benchmark. >>> >>> -- >>> S?bastien >>> >>> 2012/5/5 xancorreu > >>> >>> Hi, >>> >>> I'm just curious: can you make the tests of alioth >>> >>> [http://shootout.alioth.debian.org/u32q/which-programming-languages-are-fastest.php]? >>> If you do, you can compare pypy to cpython and also to other >>> programming languages. What do you think about this idea? >>> >>> >>> Thanks in advance, >>> Xan. >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >> >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > From santagada at gmail.com Mon May 7 00:48:01 2012 From: santagada at gmail.com (Leonardo Santagada) Date: Sun, 6 May 2012 19:48:01 -0300 Subject: [pypy-dev] alioth tests? In-Reply-To: <4FA6E933.7050103@gmail.com> References: <4FA578FF.6060304@gmail.com> <4FA584EE.6080902@gmail.com> <4FA589C6.803@gmx.de> <4FA6E933.7050103@gmail.com> Message-ID: On Sun, May 6, 2012 at 6:12 PM, xancorreu wrote: > Al 05/05/12 22:12, En/na Ronny Pfannschmidt ha escrit: > >> That got already covered some months ago, >> >> my basic understanding of the incidents that happened back then is, >> ?that alioth is not a desirable test to participate in, >> the code/scripts are biased towards particular implementations, >> and the maintainer has some strange ideas, > > > What "strange ideas" mean? that you can't use numpy for example, this hurts cpython in some of the tests. IIRC he also didn't want to warm the jit which probably hurts every language that has one. And then there is the problem that the set of benchmarks don't map too well to the kind of stuff you normally use python for. > By the other hand, saying with other words: with scripts appeared in alioth, > pypy is faster? -- Leonardo Santagada From arigo at tunes.org Mon May 7 09:41:26 2012 From: arigo at tunes.org (Armin Rigo) Date: Mon, 7 May 2012 09:41:26 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: <4FA6E933.7050103@gmail.com> References: <4FA578FF.6060304@gmail.com> <4FA584EE.6080902@gmail.com> <4FA589C6.803@gmx.de> <4FA6E933.7050103@gmail.com> Message-ID: Hi, On Sun, May 6, 2012 at 11:12 PM, xancorreu wrote: > By the other hand, saying with other words: with scripts appeared in alioth, > pypy is faster? Probably a bit, but does it matter? Consider a completely un-pythonic program that no-one would ever write this way: is the question "is PyPy faster on it" of any great relevance? The meta-answer to your question is that if we rewrite the programs to be much simpler, easier to understand, and more natural for Python (we did for a few of them), then yes, they end up being way faster on PyPy than in CPython. A bient?t, Armin. From arigo at tunes.org Mon May 7 10:01:14 2012 From: arigo at tunes.org (Armin Rigo) Date: Mon, 7 May 2012 10:01:14 +0200 Subject: [pypy-dev] feedback on PEP 421 In-Reply-To: <201205062029.q46KTbCl018261@theraft.openend.se> References: <201205062029.q46KTbCl018261@theraft.openend.se> Message-ID: Hi Laura, hi Eric, On Sun, May 6, 2012 at 10:29 PM, Laura Creighton wrote: > I think a large part of this problem is that CPython implementors are > completely unaware of what set of functions PyPy cannot implement. Indeed. From that point of view it would make sense if CPython and PyPy (to start with) co-operated in writing down a list of things in CPython that cannot be reasonably implemented on PyPy. We have already two pages listing differences, which would be a good start if it was brought to the attention of CPython developers; but it only outlines general lines, and not, I think, "sys.getrefcount() is not available on PyPy at all". http://pypy.org/compat.html http://pypy.readthedocs.org/en/latest/cpython_differences.html A bient?t, Armin. From xancorreu at gmail.com Mon May 7 15:01:44 2012 From: xancorreu at gmail.com (xancorreu) Date: Mon, 07 May 2012 15:01:44 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: References: <4FA578FF.6060304@gmail.com> <4FA584EE.6080902@gmail.com> <4FA589C6.803@gmx.de> <4FA6E933.7050103@gmail.com> Message-ID: <4FA7C7B8.3000203@gmail.com> Al 07/05/12 00:48, En/na Leonardo Santagada ha escrit: > On Sun, May 6, 2012 at 6:12 PM, xancorreu wrote: >> Al 05/05/12 22:12, En/na Ronny Pfannschmidt ha escrit: >> >>> That got already covered some months ago, >>> >>> my basic understanding of the incidents that happened back then is, >>> that alioth is not a desirable test to participate in, >>> the code/scripts are biased towards particular implementations, >>> and the maintainer has some strange ideas, >> >> What "strange ideas" mean? > that you can't use numpy for example, this hurts cpython in some of > the tests. IIRC he also didn't want to warm the jit which probably > hurts every language that has one. > > And then there is the problem that the set of benchmarks don't map too > well to the kind of stuff you normally use python for. Yeah!, I see. I think the problem is exclusion of code. Really, if benchmark among languages is made, then you should include much bunch of version the same program in every language. An exclusion is a problem because libraries are optimized for speed. Xan. > >> By the other hand, saying with other words: with scripts appeared in alioth, >> pypy is faster? > > From xancorreu at gmail.com Mon May 7 15:04:47 2012 From: xancorreu at gmail.com (xancorreu) Date: Mon, 07 May 2012 15:04:47 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: References: <4FA578FF.6060304@gmail.com> <4FA584EE.6080902@gmail.com> <4FA589C6.803@gmx.de> <4FA6E933.7050103@gmail.com> Message-ID: <4FA7C86F.7090803@gmail.com> Al 07/05/12 09:41, En/na Armin Rigo ha escrit: > Hi, > > On Sun, May 6, 2012 at 11:12 PM, xancorreu wrote: >> By the other hand, saying with other words: with scripts appeared in alioth, >> pypy is faster? > Probably a bit, but does it matter? Consider a completely un-pythonic > program that no-one would ever write this way: is the question "is > PyPy faster on it" of any great relevance? The meta-answer to your > question is that if we rewrite the programs to be much simpler, easier > to understand, and more natural for Python (we did for a few of them), > then yes, they end up being way faster on PyPy than in CPython. The meta-question is: what is the fastest language with "comfortable" syntax and expressions ;-) This is a very subjective question, because comfortable is subjective. But I think, I hope, that the benchmarks, open the programs and compare to other programs which do the same, is the only way I know to, personally, reach its answer. Thanks, Xan. > > > A bient?t, > > Armin. From arigo at tunes.org Mon May 7 15:19:23 2012 From: arigo at tunes.org (Armin Rigo) Date: Mon, 7 May 2012 15:19:23 +0200 Subject: [pypy-dev] alioth tests? In-Reply-To: <4FA7C86F.7090803@gmail.com> References: <4FA578FF.6060304@gmail.com> <4FA584EE.6080902@gmail.com> <4FA589C6.803@gmx.de> <4FA6E933.7050103@gmail.com> <4FA7C86F.7090803@gmail.com> Message-ID: Hi Xan, On Mon, May 7, 2012 at 3:04 PM, xancorreu wrote: > The meta-question is: what is the fastest language with "comfortable" syntax > and expressions ;-) Then stop looking at this particular benchmark site :-) The Python programs you see there are not in the slightest way "comfortable" at all (just like, I'm sure, the non-Python programs). Look instead for the performance running some general programs with no or minimal performance-related tweaks. It might be hard to find cross-language benchmarks doing that, but that's something we try to do on http://speed.pypy.org when comparing CPython and PyPy's speeds. A bient?t, Armin. From ericsnowcurrently at gmail.com Mon May 7 16:40:43 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 7 May 2012 08:40:43 -0600 Subject: [pypy-dev] feedback on PEP 421 In-Reply-To: References: <201205062029.q46KTbCl018261@theraft.openend.se> Message-ID: On Mon, May 7, 2012 at 2:01 AM, Armin Rigo wrote: > Hi Laura, hi Eric, > > On Sun, May 6, 2012 at 10:29 PM, Laura Creighton wrote: >> I think a large part of this problem is that CPython implementors are >> completely unaware of what set of functions PyPy cannot implement. > > Indeed. ?From that point of view it would make sense if CPython and > PyPy (to start with) co-operated in writing down a list of things in > CPython that cannot be reasonably implemented on PyPy. ?We have > already two pages listing differences, which would be a good start if > it was brought to the attention of CPython developers; but it only > outlines general lines, and not, I think, "sys.getrefcount() is not > available on PyPy at all". > > http://pypy.org/compat.html > http://pypy.readthedocs.org/en/latest/cpython_differences.html > > > A bient?t, > > Armin. This is a great point from both of you. It'll be worth pushing this forward, which I'll work on once I get sys.implementation sorted out. Keep in mind that I'm not a CPython committer, but that shouldn't matter much for pursuing this. Anyway, the idea of cataloging incompatibilities ties in pretty closely with sys.implementation and the PEP. I'll add a separate note there on how it fits into the bigger picture of making Python less CPython-specific. Any other feedback on the PEP? I'm particularly interested in your thoughts on the data structure for sys.implementation, the format for the version attribute, and the concept of the metadata attribute. Thanks! -eric From andrewfr_ice at yahoo.com Mon May 7 17:50:03 2012 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Mon, 7 May 2012 08:50:03 -0700 (PDT) Subject: [pypy-dev] Question about stm_descriptor_init(), tasklets and OS threads References: <1334616292.8656.YahooMailNeo@web120705.mail.ne1.yahoo.com> <1334857414.94198.YahooMailNeo@web120705.mail.ne1.yahoo.com> Message-ID: <1336405803.1383.YahooMailNeo@web120701.mail.ne1.yahoo.com> Hi Armin: Thanks for the explanation. Sorry for taking so long to respond. This is a challenging post. Comments below: ________________________________ From: Armin Rigo To: Andrew Francis Cc: Py Py Developer Mailing List Sent: Thursday, April 19, 2012 3:37 PM Subject: Re: Question about stm_descriptor_init(), tasklets and OS threads Hi Andrew, On Thu, Apr 19, 2012 at 19:43, Andrew Francis wrote: AF> I am trying to understand enough to get into a position to attempt an AF> integration. >I believe you are trying to approach the problem from the bottom-most >level up --- which is a fine way to approach problems; but in this >case, you are missing that there are still a few levels between were >you got so far and the destination, which is the stackless.py module >in pure Python. Yes Armin, you are right: I am not treating the transaction module as a black-box. I'll try to think more abstractly. >Let us try to approach it top-down instead, because there are far less >levels to dig through.? The plan is to make any existing >stackless-using program work on multiple cores.? Take a random >existing stackless example, and start to work by editing the pure >Python lib_pypy/stackless.py (or, at first, writing a new version from >scratch, copying parts of the original).? The idea is to use the >"transaction" module.? The goal would be to not use the _squeue, which >is a deque of pending tasklets, but instead to add pending tasklets >with transaction.add(callback).? Note how the notion of tasklets in >the _squeue (which offers some specific order) is replaced by the >notion of which callbacks have been added.? See below for what is in >each callback. >The transaction.run() would occur in the main program, directly called >by stackless.schedule().? So the callbacks are all invoked in the main >program too; in fact all the "transaction" dispatching is done in the >main program, and only scheduling new tasklets can occur anywhere. I see the transaction module and the scheduler's operation to be orthogonal. It has been my contention that the Stackless scheduler (at least in cooperative mode) is a rather dumb? mechanism. The scheduler is dumb in the sense that there really isn't much scheduling priority smarts in it. Rather scheduling/priority decisions resides in the tasklet and the channels. (Some side notes: 1) in the original Plan9 literature, the scheduler is opaque. 2) View the scheduler as being re-entrant rather than an distinct process.) This division actually makes Stackless quite flexible. For instance, when I implemented select(),? a new mechanism was introduced: a chanop (this comes straight out of the Plan9/Go implementation). Later, chanops were generalized into guards. Guards serve as the building blocks complex mechanisms such as channels and join patterns. The scheduler was not touched. Also join patterns implement a trivial form of all-or-nothing semantics (but there is no retry). Based on what you have said, I strongly suspect the transaction module could be integrated into Stackless (via stackless.py) in a similar fashion. In short, one does not touch the scheduler. However the transaction module would use the scheduler. >The callbacks would just contain a switch to the tasklet.? When the >tasklet comes back, if it is not finished, re-add() it.? This is all. Side note - in a way,? this is how the non-preemptive scheduler works. However this is C based Stackless Python. >You have to make sure that all tasklet.switch()es internally go back >to the main program, and not directly to another tasklet.? This should >ensure that the duration of every transaction is exactly the time in a >tasklet between two calls to switch(). I am having problems understanding this passage. Perhaps I view transactions and context switching? as orthogonal. Yes the transaction module's control of context switching? could be a part of? a strategy for ensuring atomicity. Yes context switching creates opportunities for conflict/contention (this is what causes race conditions). However SME schemes can handle OS context switches. >You have to make sure that all tasklet.switch()es internally go back >to the main program, and not directly to another tasklet. What is the main programme doing that makes it so special? Is it some sort of synchronizer (a la Nancy Lynch's IO Automata model)? As a side note, I recall reading in previous posts that the underlying ability to jump directly to a tasklet makes pickling difficult. Also doesn't a main programme sort of suggests a generator/trampoline approach to scheduling that wasn't necessary with tasklets and tasklets running over greenlets? Maybe it is the way yield() is used in AME that causing me a bit of mental confusion? >This should ensure that the duration of every transaction is exactly the time in a >tasklet between two calls to switch(). This implies that context switches demarcate transaction boundaries. Context switches in Stackless are very common. For instance channel operations often trigger a context switch. Programmes voluntarily call schedule() to give other tasklets a chance to execute. So in the case of: def transfer(toAccount, fromAccount, amount): ???? # start of transaction ???? fromAccount.withdraw(amount) ???? stackless.schedule() ???? toAccount.deposit(amount) ??? #finish transaction What would happen? In a normal Stackless Python programme, all other things being equal, the stackless.schedule() opens up the door for race conditions and contention. Remove the stackless.schedule() and that tasklet would operate very much like a transaction. I would suspect that if there was a transaction module present, in case of contention, it would about and re-try transfer(). Am I right in assuming this? if we could imagine transfer() under the hood doing something like: def transfer(toAccount, fromAccount, amount): ???? transaction.start()??? # NOTE THIS IS INVISIBLE TO THE PROGRAMMER!!! it would not be difficult for the transaction module, if it sees another transaction in progress that would cause contention, to block transfer() by calling say, schedule_remove() and readding transfer() when appropriate. >Of course, this can all be written and tested against >"transaction.py", the pure Python emulator.? Once it is nicely >working, you are done.? You just have to wait for a >continuation-capable version of pypy-stm; running the same program on >it, you'll get multi-core usage. Once again, thank you for your response. I may be still unsure of? some aspects. However I believe integration ought to be much cleaner - no hacking/replacing the scheduler. Cheers, Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: From gbowyer at fastmail.co.uk Mon May 7 21:30:30 2012 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Mon, 07 May 2012 12:30:30 -0700 Subject: [pypy-dev] Biting off more than I can chew In-Reply-To: References: <4F42DB6C.7010307@fastmail.co.uk> <4F47FFD2.8020600@fastmail.co.uk> Message-ID: <4FA822D6.7080305@fastmail.co.uk> It has taken me a long time to get back to this, the algorithm I am playing with is soft real-time at best not hard realtime My thinking is far more around using (and abusing) the transaction queue to manage a barrier for the GC implementing much of a classical read/write barrier there On 25/02/12 09:42, Armin Rigo wrote: > Re-hi, > > Ah, another matter. I don't know how interesting the goal of a > no-pause GC is together with STM. It would seem to me that STM is, by > itself, not really well-suited if you want to absolutely avoid pauses: > if a transaction is aborted a few times before eventually succeeding > to commit, then you have an unexpected pause between the time were the > transaction was first started and the time were it committed. I think > it's a built-in property of all transactional memory systems to not be > really "real-time safe". Maybe they can be made real-time safe by > falling back to nontransactional single-threaded execution at some > point, but that looks more advanced that I care for right now. > > > A bient?t, > > Armin. From arigo at tunes.org Tue May 8 11:46:15 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 8 May 2012 11:46:15 +0200 Subject: [pypy-dev] cppyy: C++ bindings for PyPy In-Reply-To: References: Message-ID: Hi Wim, On Fri, May 4, 2012 at 8:16 AM, wrote: >> Quick question - if it's mature, why not merge it to default? I presume it >> should be turned off, since there is a sizeable dependency, but still >> having it in default can be good. > > I have a standalone version of Reflex now and made a few minor mods to cppyy > to use it. (...) I think that Fijal's suggestion was to merge the branch anyway, but keep cppyy turned off by default. I would also welcome this, but it's up to you if you prefer to keep it in a separate branch. I can give a try to lazy linking, but I fear that it's going to be OS-specific code... A bient?t, Armin. From printingpub at 126.com Tue May 8 13:35:24 2012 From: printingpub at 126.com (admin hc) Date: Tue, 08 May 2012 11:35:24 +0000 Subject: [pypy-dev] =?gb2312?b?YWRtaW4gaGPT68T6ubLP7cHL1dXGrA==?= Message-ID: <047d7b33d98a1c587604bf846e6f@google.com> Dear Manager, Good day, my friend. Thanks for your time to read my email. This is Sandy from H&C Printing Center Ltd in Xiamen, China. From your company web, we got to know that your company is an influential company in your business. We are pleased to approach you in the hope of developing business for mutual benefit. Our products services are as follows: 1. Products: Magazine, Catalog, Picture Album, Book, Booklet, Calendar etc. (You can tell us other printing products you need.) 2. Transportation: Direct vessels to many international ports. 3. Printing Price: Only 40% cost of your local printers?? 4. Period of manufacturing: Effective and quick. 5. Advantage: Perfect Quality, Competitive Prices, Good Services. Wish our products will be helpful for your business. Any questions, feel free to contact us by admin at hongchengco.com . I am looking forward to your prompt response. Sincerely, Sandy -------------- next part -------------- A non-text attachment was scrubbed... Name: printingcenter.jpg Type: image/jpeg Size: 21271 bytes Desc: not available URL: From arigo at tunes.org Tue May 8 14:18:16 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 8 May 2012 14:18:16 +0200 Subject: [pypy-dev] Mini-sprint next week in Leysin (14-19th of May) Message-ID: Hi all, With Maciej showing up, we will have "just in time" a mini-sprint next week in Leysin: 14-19th of May. Anyone is welcome to show up, from a day to a week, to talk or work on anything preferrably vaguely PyPy-related. As usual the topics are open, but we were thinking at least to implement some optimization in the JIT: allocating the assembler frames into the heap rather than the stack. (This *is* an optimization because they can survive even after the function returned, which means that we don't need to move parts of them out when the function returns.) If someone wants to jump on the offer, mail me so that we can set up logistics. A bient?t, Armin. From ugear-g777 at umail.hinet.net Tue May 8 17:11:58 2012 From: ugear-g777 at umail.hinet.net (=?utf-8?B?VSBHZWFyIEdyb3Vw?=) Date: 8 May 2012 23:11:58 +0800 Subject: [pypy-dev] =?utf-8?b?UFZDIENvbXBvdW5kIEBVUyQwLjg4L0vigItHKio=?= Message-ID: <201205081511.q48F03o0028614@msr14.hinet.net> An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Wed May 9 10:16:03 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 9 May 2012 01:16:03 -0700 (PDT) Subject: [pypy-dev] cppyy: C++ bindings for PyPy In-Reply-To: References: Message-ID: Hi Armin, > I think that Fijal's suggestion was to merge the branch anyway, but > keep cppyy turned off by default. I would also welcome this, but it's > up to you if you prefer to keep it in a separate branch. I feel better about merging now that I've created a standalone version of Reflex. However, I'll be mostly offline for the next two weeks or so, so I'll wait until after that, to be able to deal with possible fall-out. :} > I can give a try to lazy linking, but I fear that it's going to be > OS-specific code... That's in any case a separate can of worms, as nothing has yet been tested on Windows (the only Windows box that I have, has only 2GB of memory, so too little to translate, although I could try to run the tests). Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From arigo at tunes.org Wed May 9 16:01:24 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 9 May 2012 16:01:24 +0200 Subject: [pypy-dev] pypy-stm Message-ID: Hi all, Here is the very first Python 2.7 interpreter to run existing multithreaded programs on multiple cores. Linux 32 bits. It is horribly slow right now, so consider this as nothing more than a demo :-) http://wyvern.cs.uni-duesseldorf.de/~arigo/pypy-c-r54960-stm.tar.bz2 Documentation: https://bitbucket.org/pypy/pypy/raw/stm-thread/pypy/doc/stm.rst A bient?t, Armin. From dynamicgl at gmail.com Wed May 9 16:23:14 2012 From: dynamicgl at gmail.com (gelin yan) Date: Wed, 9 May 2012 22:23:14 +0800 Subject: [pypy-dev] pypy-stm In-Reply-To: References: Message-ID: On Wed, May 9, 2012 at 10:01 PM, Armin Rigo wrote: > Hi all, > > Here is the very first Python 2.7 interpreter to run existing > multithreaded programs on multiple cores. Linux 32 bits. It is > horribly slow right now, so consider this as nothing more than a demo > :-) > > http://wyvern.cs.uni-duesseldorf.de/~arigo/pypy-c-r54960-stm.tar.bz2 > > Documentation: > > https://bitbucket.org/pypy/pypy/raw/stm-thread/pypy/doc/stm.rst > > > A bient?t, > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > It is great...Anyway the ball is starting to roll... -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddiejacobs at caryon.com Wed May 9 21:21:24 2012 From: freddiejacobs at caryon.com (Freddie Jacobsen) Date: Wed, 9 May 2012 20:21:24 +0100 Subject: [pypy-dev] Web Place/ Analysis/ Review Message-ID: <65361378.20120509202124@caryon.com> pypy-dev at codespeak.net : Your site has a good look and feel but you're not getting as much web traffic as you could be getting. Can we show you how to change that? It won't cost you anything for us to review your online business and the results could spike your web revenue. Reply to us today and we will do a free assessment of your web presence. Don't forget to include the best way we can contact you. Sincerely, Freddie Jacobsen -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 10 09:41:30 2012 From: arigo at tunes.org (Armin Rigo) Date: Thu, 10 May 2012 09:41:30 +0200 Subject: [pypy-dev] Question about stm_descriptor_init(), tasklets and OS threads In-Reply-To: <1336405803.1383.YahooMailNeo@web120701.mail.ne1.yahoo.com> References: <1334616292.8656.YahooMailNeo@web120705.mail.ne1.yahoo.com> <1334857414.94198.YahooMailNeo@web120705.mail.ne1.yahoo.com> <1336405803.1383.YahooMailNeo@web120701.mail.ne1.yahoo.com> Message-ID: Hi Andrew, Please look at the latest documentation: https://bitbucket.org/pypy/pypy/raw/stm-thread/pypy/doc/stm.rst You should be able to use such a "thread.atomic" in stackless.py. You need to create N threads and run the tasklets in these threads. As long as each tasklet's user code is protected by a "thread.atomic", then they will *appear* to be run serially. You probably need to call "thread.atomic.__enter__" and "__exit__" explicitly for your use case; if you do, then I could also move the functionality as normal built-in method. You also have to handle issues like tasklets not being always allowed to switch threads. As a first approximation, on CPython you can implement a dummy "thread.atomic" by acquiring and releasing a single lock. It is only an approximative equivalent because other non-atomic threads will be allowed to run concurrently; but for this kind of experiment where *all* threads should be "atomic", it should not make a difference. A bient?t, Armin. From fijall at gmail.com Thu May 10 11:19:58 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 10 May 2012 11:19:58 +0200 Subject: [pypy-dev] cppyy: C++ bindings for PyPy In-Reply-To: References: Message-ID: On Wed, May 9, 2012 at 10:16 AM, wrote: > Hi Armin, > > > I think that Fijal's suggestion was to merge the branch anyway, but >> keep cppyy turned off by default. I would also welcome this, but it's >> up to you if you prefer to keep it in a separate branch. >> > > I feel better about merging now that I've created a standalone version of > Reflex. However, I'll be mostly offline for the next two weeks or so, so > I'll > wait until after that, to be able to deal with possible fall-out. :} There is absolutely no rush, however great news! Can you think about a blogpost describing what you did and how to use it accompanying the merge? Copy-pasting docs is fine :) Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 10 12:49:31 2012 From: arigo at tunes.org (Armin Rigo) Date: Thu, 10 May 2012 12:49:31 +0200 Subject: [pypy-dev] pypy-stm In-Reply-To: References: Message-ID: Re-hi, I updated my fork of CPython 3.x to include a very similar functionality, _thread.atomic_enter() and _thread.atomic_exit(). (Of course without any multicore capabilities, but it gives you the atomicity.) A very small diff overall. http://bitbucket.org/arigo/cpython-withatomic A bient?t, Armin. From wlavrijsen at lbl.gov Thu May 10 22:48:55 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 10 May 2012 13:48:55 -0700 (PDT) Subject: [pypy-dev] cppyy: C++ bindings for PyPy In-Reply-To: References: Message-ID: Hi Maciej, > Can you think about a blogpost describing what you did and how to use it > accompanying the merge? yes, that was the intention: I had actually already started drafting a blog post, but got distracted. For the how-to, I'd link directly to the docs: http://doc.pypy.org/en/latest/cppyy.html as there's more room there for detailed instructions. In the posting, I'd rather focus on explaining the decisions that went into the development. Cheers, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From newsletter at ensinfo.org Fri May 11 11:41:45 2012 From: newsletter at ensinfo.org (ENS Newsletter) Date: Fri, 11 May 2012 11:41:45 +0200 Subject: [pypy-dev] Unsubscribe Newsletter ENS Message-ID: <4f5e8ebd-2166-4425-984e-9e3302840dcb@CEN-SV-EXM-02.congrex.internal> An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri May 11 12:31:17 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 11 May 2012 12:31:17 +0200 Subject: [pypy-dev] pypy-stm In-Reply-To: References: Message-ID: Hi again, An update of pypy-stm. The previous one was not working properly with any Python function call --- on any example of any non-trivial size, it would not actually run on multiple cores. This one works better. http://wyvern.cs.uni-duesseldorf.de/~arigo/pypy-c-r55013-stm.tar.bz2 I also did a first pure Python version of 'transaction.py', which you can find either from the stm-thread branch in lib_pypy/, or directly from https://bitbucket.org/pypy/pypy/raw/stm-thread/lib_pypy/transaction.py . The next step is to add some logging mecanism that should tell you more precisely why transactions become inevitable, e.g. sorted by the total amount of time spent running an atomic block in inevitable mode. A bient?t, Armin. From residual at yahoo.com Sat May 12 08:11:26 2012 From: residual at yahoo.com (Shirley) Date: 12 May 2012 08:11:26 +0200 Subject: [pypy-dev] Your friend Shirley has recommended this great product from Misjah's Shop Message-ID: <20120512061126.13369.qmail@ds11.beijen.com> Hi marketer! Your friend, Shirley, thought that you would be interested in DJ Misjah vs Kouichi Okamoto from Misjah's Shop. Hey I need your help to beta-test this amazing new system that's already resulted in profits of $558,087 in cold hard cash... If you've never made a penny online before, and you want to be able to create a profit pulling machine in just 7 clicks of your mouse http://dollaraffmachine.co.cc/aff/item.php?usn=dap1&i=it_ep&e=pypy-dev at codespeak.net - You don't need an existing website - You don't need any technical experience - You don't need to be a sales person - You don't need any money to get started - You don't need to learn about SEO, Twitter, link-building, CPA, PPV or any of that - You don't need to spend hours of your time each day to make any money In fact, beginners will get preference http://dollaraffmachine.co.cc/aff/item.php?usn=dap1&i=it_ep&e=pypy-dev at codespeak.net Application Requirements: - Provide feedback in terms of the features and functionality of the product - Let us know how fat your bank account has become http://dollaraffmachine.co.cc/aff/item.php?usn=dap1&i=it_ep&e=pypy-dev at codespeak.net ACT NOW only 300 beta-testers are required and the closing date is fast approaching Thanks Residual Group PS this guy will also explain to you why it's a simple choice - cash in... or go to jail! You decide: http://dollaraffmachine.co.cc/aff/item.php?usn=dap1&i=it_ep&e=pypy-dev at codespeak.net To unsubscribe visit: http://dollaraffmachine.co.cc/un.php?e=pypy-dev at codespeak.net . To view the product click on the link below or copy and paste the link into your web browser: http://www.djmisjah.com/catalog/product_info.php?products_id=108 Regards, Misjah's Shop http://www.djmisjah.com/catalog/ From benjamin at python.org Tue May 15 04:08:41 2012 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 14 May 2012 19:08:41 -0700 Subject: [pypy-dev] AOSA book chapter Message-ID: Hello pypy-dev, As some of you may know, this past year I wrote a chapter about PyPy for Architecture of Open Source Applications Volume 2. It has brought to my attention that my communication with the PyPy community about it was poor. Since it deeply involved the PyPy project, I should have been more public in its writing; I should have sent it to pypy-dev for review and comments. Now that this has been pointed out to me, it's painfully obvious that I fell short. Next time such an opportunity arises, I look forward to working fully with the PyPy community on it. Please accept my apologies, Benjamin From alex.gaynor at gmail.com Tue May 15 04:11:13 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 14 May 2012 22:11:13 -0400 Subject: [pypy-dev] AOSA book chapter In-Reply-To: References: Message-ID: On Mon, May 14, 2012 at 10:08 PM, Benjamin Peterson wrote: > Hello pypy-dev, > As some of you may know, this past year I wrote a chapter about PyPy > for Architecture of Open Source Applications Volume 2. It has brought > to my attention that my communication with the PyPy community about it > was poor. Since it deeply involved the PyPy project, I should have > been more public in its writing; I should have sent it to pypy-dev for > review and comments. Now that this has been pointed out to me, it's > painfully obvious that I fell short. > > Next time such an opportunity arises, I look forward to working fully > with the PyPy community on it. > > Please accept my apologies, > Benjamin > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Hey Benjamin, Without wanting to step on anyone's toes, I don't think you had any obligation to notify us. I haven't read your chapter, but I look forward to it and the rest of AOSA. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Tue May 15 15:07:51 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 15 May 2012 15:07:51 +0200 Subject: [pypy-dev] Documenting merged branches Message-ID: <4FB25527.40602@gmail.com> Hi all, the most laborious part of doing a new release is writing the "What's new" document. Together with Armin and Maciek, we tried to find a way to make it easier and less time consuming. The idea is that whenever someone merges a branch to default, he should also briefly document what it does by writing a couple of lines in doc/whatsnew-1.9.rst. I also added a test (doc/test/test_whatsnew.py) which checks that all the merged branches are mentioned in the document. To document a branch, you have to include a special rst comment ".. branch: BRANCH-NAME" which is recognized by the test. If the branch is uninteresting for the what's new document, one can just add the comment somewhere in the doc without writing anything. At the moment the test is passing because I added empty docs for all the branches merged so far. It would be nice if everyone could have a look at it and document the branches he merged. ciao, Anto From cfbolz at gmx.de Tue May 15 15:35:10 2012 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 15 May 2012 15:35:10 +0200 Subject: [pypy-dev] AOSA book chapter In-Reply-To: References: Message-ID: Hi all, On Tue, May 15, 2012 at 4:11 AM, Alex Gaynor wrote: > On Mon, May 14, 2012 at 10:08 PM, Benjamin Peterson > wrote: >> As some of you may know, this past year I wrote a chapter about PyPy >> for Architecture of Open Source Applications Volume 2. It has brought >> to my attention that my communication with the PyPy community about it >> was poor. Since it deeply involved the PyPy project, I should have >> been more public in its writing; I should have sent it to pypy-dev for >> review and comments. Now that this has been pointed out to me, it's >> painfully obvious that I fell short. >> >> Next time such an opportunity arises, I look forward to working fully >> with the PyPy community on it. >> >> Please accept my apologies, Accepted, thank you for writing this mail. > Without wanting to step on anyone's toes, I don't think you had any > obligation to notify us.? I haven't read your chapter, but I look forward to > it and the rest of AOSA. I agree, no obligation. But it would still have been much fairer to do it, given that both open source as well as science work based on reputation. Cheers, Carl Friedrich From arigo at tunes.org Tue May 15 16:23:14 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 15 May 2012 16:23:14 +0200 Subject: [pypy-dev] Python FFI Message-ID: Hi all, Fijal and me would like to raise interest among various groups of people about building a better ctypes replacement for Python. The general background first, at least as far as we know it. People generally agree that CPython extension modules are not extremely hard to write, but also not the ultimate solution in the matter of interfacing with C code. Obviously you need to know and write C code, and learn the large CPython C API. Moreover a CPython extension module is rather specific to CPython or even a particular version of it. Even if you are lucky and your CPython extensions work on all 2.x versions, it won't work unmodified on Python 3.x, and be slow (or not work at all) on PyPy. (I'm leaving Jython and IronPython out of the picture here but it's very far from ideal for them too.) The working alternatives nowadays are Cython and ctypes. For Cython you have to learn a whole new language; we can argue infinitely about preferences, but from my point of view it's mostly just a nicer way to write CPython C extensions using --- instead of C --- a custom language. About ctypes, I would say that the fact that its use remains marginal is enough to show that its mission kind of failed. Its problems are more than one, but most importantly the level at which you describe the information is slightly too low - it's ABI-level instead of API-level. It does not make a difference for some libraries, but for some it does make a huge difference, like when you use macros. We would like something that can be potentially used to wrap almost all C libraries. So we would like to propose something (a bit more publically than the two subprojects of PyPy attempting to do that --- did you know about their existence?). The goal would be to come up with a better-designed project. "Better designed" in my vocabulary implies "simpler". I'm starting in this mailing list but the idea is to try to move it in its own project, and to develop at the same time at least a CPython 2.7 and 3.x and PyPy implementation. The simplest FFI we know of for a high-level language is LuaJIT's FFI. If you are interested, read the four pages starting at http://luajit.org/ext_ffi.html . A crucial design decision was to avoid making use of tons of objects representing the C types; instead, the user mostly specifies C types as plain strings, simplifying the basic API a lot. So we would like to propose some design very similar to that one. The basic public API looks more or less fine to me. There are of course preferences and topics to discuss, but let's leave that for later. Most importantly, we would welcome a system where the C declarations can be parsed either internally or (for more complicated cases) externally, with the help of a real C compiler. This might include cases like macros, where the user would need to write (as C code) a small function calling the macro, and have it compiled when he imports his Python module. Also, because it would parse strings, we can also add support for parsing strings in the format of Cython declarations. This would still lower the entry barrier, and have the nice effect that the user can sometimes copy-and-paste Cython declarations, instead of copy-and-pasting carefully chosen pieces of a potentially messy .h file full of custom macros (see for example /usr/include/zlib.h). The benefit would be that, unlike Cython, he gets to call the declared functions directly from his regular Python code. Opinions? Interests? This mail is deliberately low on details about how we think we can do it. Instead I'm seeking general reactions for now and would like to move this soon to its own project, independent of PyPy. A bient?t, Armin. From arigo at tunes.org Tue May 15 16:29:31 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 15 May 2012 16:29:31 +0200 Subject: [pypy-dev] AOSA book chapter In-Reply-To: References: Message-ID: Hi Benjamin, On Tue, May 15, 2012 at 4:08 AM, Benjamin Peterson wrote: > As some of you may know, this past year I wrote a chapter about PyPy > for Architecture of Open Source Applications Volume 2. Great, congratulation :-) > It has brought to my attention that my communication with the PyPy community about it > was poor. Don't worry about it. Armin From holger at merlinux.eu Tue May 15 17:01:51 2012 From: holger at merlinux.eu (holger krekel) Date: Tue, 15 May 2012 15:01:51 +0000 Subject: [pypy-dev] AOSA book chapter In-Reply-To: References: Message-ID: <20120515150151.GA10174@merlinux.eu> Hi Benjamin, On Mon, May 14, 2012 at 19:08 -0700, Benjamin Peterson wrote: > Hello pypy-dev, > As some of you may know, this past year I wrote a chapter about PyPy > for Architecture of Open Source Applications Volume 2. It has brought > to my attention that my communication with the PyPy community about it > was poor. Since it deeply involved the PyPy project, I should have > been more public in its writing; I should have sent it to pypy-dev for > review and comments. Now that this has been pointed out to me, it's > painfully obvious that I fell short. > > Next time such an opportunity arises, I look forward to working fully > with the PyPy community on it. I quick-read the chapter and like it. Congrats! Probably makes sense to link to it from pypy docs. I also think it's a good idea to share information about such endeavours early and especially if a third party intends to talk to the "pypy" project as whole (no clue if that was the case here). best, holger > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Tue May 15 17:11:52 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 15 May 2012 17:11:52 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: <4FB27238.5020509@gmail.com> Hi Armin, hi all, On 05/15/2012 04:23 PM, Armin Rigo wrote: > Hi all, > > Fijal and me would like to raise interest among various groups of > people about building a better ctypes replacement for Python. [cut] > Opinions? Interests? This mail is deliberately low on details about > how we think we can do it. Instead I'm seeking general reactions for > now and would like to move this soon to its own project, independent > of PyPy. since there are few details there is not much to discuss apart general interest. As you know I've also been having the idea of doing something similar for a while, so I'd certainly be interested in getting involved as soon as the discussion moves to its "new place". ciao, Anto From skip at pobox.com Tue May 15 17:14:54 2012 From: skip at pobox.com (Skip Montanaro) Date: Tue, 15 May 2012 10:14:54 -0500 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: > So we would like to propose something (a bit more publically than the > two subprojects of PyPy attempting to do that --- did you know about > their existence?). No, can you provide references? > The simplest FFI we know of for a high-level language is LuaJIT's FFI. > ?If you are interested, read the four pages starting at > http://luajit.org/ext_ffi.html.... > So we would like to propose some design very similar to that one.... > Opinions? ?Interests? ?This mail is deliberately low on details about > how we think we can do it. ?Instead I'm seeking general reactions for > now and would like to move this soon to its own project, independent > of PyPy. I'm no great fan of C++, but work in an environment where C++ support is a must. (We currently used Boost.Python.) The Lua docs didn't mention C++. Do either of the PyPy subprojects you mentioned address C++ interfaces? Do you plan to support it? Thx, Skip From stefan_ml at behnel.de Tue May 15 17:16:52 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 May 2012 17:16:52 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: Armin Rigo, 15.05.2012 16:23: > Fijal and me would like to raise interest among various groups of > people about building a better ctypes replacement for Python. > [...] > The working alternatives nowadays are Cython and ctypes. For Cython > you have to learn a whole new language; we can argue infinitely about > preferences, but from my point of view it's mostly just a nicer way to > write CPython C extensions using --- instead of C --- a custom > language. Well, it's basically Python with optional static types, which are mostly provided in C(-like) notation. I find it surprisingly similar to LuaJIT's FFI, since you mentioned that anyway. > Also, because it would parse strings, we can also add support for > parsing strings in the format of Cython declarations. This would > still lower the entry barrier, and have the nice effect that the user > can sometimes copy-and-paste Cython declarations, instead of > copy-and-pasting carefully chosen pieces of a potentially messy .h > file full of custom macros (see for example /usr/include/zlib.h). The > benefit would be that, unlike Cython, he gets to call the declared > functions directly from his regular Python code. I assume you are aware of Cython's pure Python mode? It doesn't currently support calling into C, but that's a matter of doing it, not so much of researching how it could be done. Personally, I've always been thinking of Cython as the best FFI that Python could ever have. Why reinvent the wheel? Do you really think you could come up with something as powerful as Cython, but substantially simpler? Stefan From stefan_ml at behnel.de Tue May 15 17:30:09 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 May 2012 17:30:09 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: Skip Montanaro, 15.05.2012 17:14: >> So we would like to propose something (a bit more publically than the >> two subprojects of PyPy attempting to do that --- did you know about >> their existence?). > > No, can you provide references? > >> The simplest FFI we know of for a high-level language is LuaJIT's FFI. >> If you are interested, read the four pages starting at >> http://luajit.org/ext_ffi.html.... > >> So we would like to propose some design very similar to that one.... > >> Opinions? Interests? This mail is deliberately low on details about >> how we think we can do it. Instead I'm seeking general reactions for >> now and would like to move this soon to its own project, independent >> of PyPy. > > I'm no great fan of C++, but work in an environment where C++ support > is a must. (We currently used Boost.Python.) The Lua docs didn't > mention C++. Do either of the PyPy subprojects you mentioned address > C++ interfaces? Do you plan to support it? AFAIR, C++ can't be connected through a ctypes-like call time API per-se, because it lacks a well defined ABI. It could be done through runtime generation of interconnection code, although that involves some dependencies. Stefan From fijall at gmail.com Tue May 15 17:39:54 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 15 May 2012 17:39:54 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: On Tue, May 15, 2012 at 5:16 PM, Stefan Behnel wrote: > Armin Rigo, 15.05.2012 16:23: > > Fijal and me would like to raise interest among various groups of > > people about building a better ctypes replacement for Python. > > [...] > > The working alternatives nowadays are Cython and ctypes. For Cython > > you have to learn a whole new language; we can argue infinitely about > > preferences, but from my point of view it's mostly just a nicer way to > > write CPython C extensions using --- instead of C --- a custom > > language. > > Well, it's basically Python with optional static types, which are mostly > provided in C(-like) notation. I find it surprisingly similar to LuaJIT's > FFI, since you mentioned that anyway. > > > > Also, because it would parse strings, we can also add support for > > parsing strings in the format of Cython declarations. This would > > still lower the entry barrier, and have the nice effect that the user > > can sometimes copy-and-paste Cython declarations, instead of > > copy-and-pasting carefully chosen pieces of a potentially messy .h > > file full of custom macros (see for example /usr/include/zlib.h). The > > benefit would be that, unlike Cython, he gets to call the declared > > functions directly from his regular Python code. > > I assume you are aware of Cython's pure Python mode? It doesn't currently > support calling into C, but that's a matter of doing it, not so much of > researching how it could be done. > > Personally, I've always been thinking of Cython as the best FFI that Python > could ever have. Why reinvent the wheel? Do you really think you could come > up with something as powerful as Cython, but substantially simpler? > > Stefan Hi Stefan. Cython's pure-python mode requires quite a bit of work to be able to work and design I think, although feel free to prove me wrong. Most importantly, it's not only about calls - cython deals with it just fine. I want to be able to manipulate the resulting data structures and arguments from pure python and do all of that without compiling anything. This is part one, however part two is that you need some extra step - you need at the very least preprocessing that would remove stuff from files that's incorrect python, if I understood you correctly. This is quite different from what I have in mind where you don't have to have an extra compile/preprocess step anywhere. Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue May 15 17:41:07 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 15 May 2012 17:41:07 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: Hi Skip, On Tue, May 15, 2012 at 5:14 PM, Skip Montanaro wrote: > Do either of the PyPy subprojects you mentioned address > C++ interfaces? ?Do you plan to support it? No. The subprojects I mentioned are the "ffistruct" branch introducing a new built-in module by Antonio, and a different design document by Alex Gaynor (I don't even remember where?..). These two and the LuaJIT FFI are about C, not C++. The C++ side is being worked on from a different angle by Wim; see recent pypy-dev mails about cppyy. It's an interface that requires in all cases a C++ compiler, which I imagine is completely unavoidable for C++. A bient?t, Armin. From yselivanov.ml at gmail.com Tue May 15 17:56:55 2012 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 15 May 2012 11:56:55 -0400 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: <226AED52-21D7-4834-A2B9-D9DDB10C4AF7@gmail.com> On 2012-05-15, at 10:23 AM, Armin Rigo wrote: > Opinions? Interests? This mail is deliberately low on details about > how we think we can do it. Instead I'm seeking general reactions for > now and would like to move this soon to its own project, independent > of PyPy. While I don't like some design quirks of Cython, I think that it's far better than any ffi or ctypes-like solution. Essentially, it's an ffi merged with the language, not a separate module. And that's a pretty unique approach. Why not focus on improving Cython design and moving toward some superset of Python language, that every implementation should support natively? - Yury From stefan_ml at behnel.de Tue May 15 18:14:19 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 May 2012 18:14:19 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: Maciej Fijalkowski, 15.05.2012 17:39: > On Tue, May 15, 2012 at 5:16 PM, Stefan Behnel wrote: >> Armin Rigo, 15.05.2012 16:23: >>> Fijal and me would like to raise interest among various groups of >>> people about building a better ctypes replacement for Python. >>> [...] >>> The working alternatives nowadays are Cython and ctypes. For Cython >>> you have to learn a whole new language; we can argue infinitely about >>> preferences, but from my point of view it's mostly just a nicer way to >>> write CPython C extensions using --- instead of C --- a custom >>> language. >> >> Well, it's basically Python with optional static types, which are mostly >> provided in C(-like) notation. I find it surprisingly similar to LuaJIT's >> FFI, since you mentioned that anyway. >> >> >>> Also, because it would parse strings, we can also add support for >>> parsing strings in the format of Cython declarations. This would >>> still lower the entry barrier, and have the nice effect that the user >>> can sometimes copy-and-paste Cython declarations, instead of >>> copy-and-pasting carefully chosen pieces of a potentially messy .h >>> file full of custom macros (see for example /usr/include/zlib.h). The >>> benefit would be that, unlike Cython, he gets to call the declared >>> functions directly from his regular Python code. >> >> I assume you are aware of Cython's pure Python mode? It doesn't currently >> support calling into C, but that's a matter of doing it, not so much of >> researching how it could be done. >> >> Personally, I've always been thinking of Cython as the best FFI that Python >> could ever have. Why reinvent the wheel? Do you really think you could come >> up with something as powerful as Cython, but substantially simpler? > > Cython's pure-python mode requires quite a bit of work to be able to work > and design I think, although feel free to prove me wrong. As I said, it lacks a way to use external C declarations. This could be done by enabling cimports to read these declarations from an external .pxd file, or, less nicely, because it won't play well with editors: read them from a string. I'm not a fan of relying on parsing header files directly, because It Will Not Work. And it it works, it will Not Do The Right Thing. > Most importantly, > it's not only about calls - cython deals with it just fine. I want to be > able to manipulate the resulting data structures and arguments from pure > python and do all of that without compiling anything. I see your point, although I think you will eventually have to compile something (or generate assembly) in one way or another, at least for C++ support. Yes, I'm well aware that a C/C++ compiler is a major dependency to carry around. However, I'm not sure how you want this to work (efficiently?) with CPython. The reason why it works in LuaJIT is that LuaJIT has a JIT compiler that can translate it down to native code. PyPy could obviously do that, too. I'm inclined to think that Jython won't be able to do it all that well, because it runs inside of a VM that requires at least a JNI call in between. No idea how IronPython would deal with it. However, CPython doesn't have a JIT compiler, and that's one of the biggest drawbacks for (something like) ctypes. There's a reason we compile real code in Cython, not just a thin glue layer. So, if you want something that's usable outside of PyPy, you will have to come up with something that's useful in CPython as well. > This is part one, > however part two is that you need some extra step - you need at the very > least preprocessing that would remove stuff from files that's incorrect > python, if I understood you correctly. This is quite different from what I > have in mind where you don't have to have an extra compile/preprocess step > anywhere. I think you want some preprocessing, if only for performance reasons. And while some CPython users may really want to call non-performance critical C stuff from time to time, you will have a hard time convincing everyone that they need to port their whole legacy environment to PyPy just to speed up their calls into external code. I'm sure they'll quite happily stick with Cython instead. Stefan From markflorisson88 at gmail.com Tue May 15 18:29:53 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 15 May 2012 17:29:53 +0100 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: On 15 May 2012 17:14, Stefan Behnel wrote: > Maciej Fijalkowski, 15.05.2012 17:39: >> On Tue, May 15, 2012 at 5:16 PM, Stefan Behnel wrote: >>> Armin Rigo, 15.05.2012 16:23: >>>> Fijal and me would like to raise interest among various groups of >>>> people about building a better ctypes replacement for Python. >>>> [...] >>>> The working alternatives nowadays are Cython and ctypes. ?For Cython >>>> you have to learn a whole new language; we can argue infinitely about >>>> preferences, but from my point of view it's mostly just a nicer way to >>>> write CPython C extensions using --- instead of C --- a custom >>>> language. >>> >>> Well, it's basically Python with optional static types, which are mostly >>> provided in C(-like) notation. I find it surprisingly similar to LuaJIT's >>> FFI, since you mentioned that anyway. >>> >>> >>>> Also, because it would parse strings, we can also add support for >>>> parsing strings in the format of Cython declarations. ?This would >>>> still lower the entry barrier, and have the nice effect that the user >>>> can sometimes copy-and-paste Cython declarations, instead of >>>> copy-and-pasting carefully chosen pieces of a potentially messy .h >>>> file full of custom macros (see for example /usr/include/zlib.h). ?The >>>> benefit would be that, unlike Cython, he gets to call the declared >>>> functions directly from his regular Python code. >>> >>> I assume you are aware of Cython's pure Python mode? It doesn't currently >>> support calling into C, but that's a matter of doing it, not so much of >>> researching how it could be done. >>> >>> Personally, I've always been thinking of Cython as the best FFI that Python >>> could ever have. Why reinvent the wheel? Do you really think you could come >>> up with something as powerful as Cython, but substantially simpler? >> >> Cython's pure-python mode requires quite a bit of work to be able to work >> and design I think, although feel free to prove me wrong. > > As I said, it lacks a way to use external C declarations. This could be > done by enabling cimports to read these declarations from an external .pxd > file, or, less nicely, because it won't play well with editors: read them > from a string. > > I'm not a fan of relying on parsing header files directly, because It Will > Not Work. And it it works, it will Not Do The Right Thing. > I think a parser may not always be able to do the right thing in case of macros etc, but I think the compiled wrapper approach will work quite well, especially since the user writes the code explicitly. >> Most importantly, >> it's not only about calls - cython deals with it just fine. I want to be >> able to manipulate the resulting data structures and arguments from pure >> python and do all of that without compiling anything. > > I see your point, although I think you will eventually have to compile > something (or generate assembly) in one way or another, at least for C++ > support. Yes, I'm well aware that a C/C++ compiler is a major dependency to > carry around. > > However, I'm not sure how you want this to work (efficiently?) with > CPython. The reason why it works in LuaJIT is that LuaJIT has a JIT > compiler that can translate it down to native code. PyPy could obviously do > that, too. I'm inclined to think that Jython won't be able to do it all > that well, because it runs inside of a VM that requires at least a JNI call > in between. No idea how IronPython would deal with it. > > However, CPython doesn't have a JIT compiler, and that's one of the biggest > drawbacks for (something like) ctypes. There's a reason we compile real > code in Cython, not just a thin glue layer. So, if you want something > that's usable outside of PyPy, you will have to come up with something > that's useful in CPython as well. > > >> This is part one, >> however part two is that you need some extra step - you need at the very >> least preprocessing that would remove stuff from files that's incorrect >> python, if I understood you correctly. This is quite different from what I >> have in mind where you don't have to have an extra compile/preprocess step >> anywhere. > > I think you want some preprocessing, if only for performance reasons. And > while some CPython users may really want to call non-performance critical C > stuff from time to time, you will have a hard time convincing everyone that > they need to port their whole legacy environment to PyPy just to speed up > their calls into external code. I'm sure they'll quite happily stick with > Cython instead. I don't think that's the point though, nor the problem they are trying to solve. CTypes is a pretty tough interface to deal with these problems, and any better interface can benefit many Python users, including the people currently using ctypes, and including people that have a simple api they need to call or just prefer portability over using Cython. It would also perhaps allow better sharing of certain stdlib modules between Python implementations (?). > Stefan > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From fijall at gmail.com Tue May 15 18:31:12 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 15 May 2012 18:31:12 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: > > > However, CPython doesn't have a JIT compiler, and that's one of the biggest > drawbacks for (something like) ctypes. There's a reason we compile real > code in Cython, not just a thin glue layer. So, if you want something > that's usable outside of PyPy, you will have to come up with something > that's useful in CPython as well. > > Completely ignoring your other issues for now. No, this is not the major drawback for ctypes for me. I know people who would happily use something that works nicely on pypy and cpython without performance considerations. The problem is that ctypes is at the wrong level - you cannot do some things and that's where cython helps. Ctypes is also crazy and inconsistent and hard to use, this makes it even more problematic. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue May 15 18:34:10 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 15 May 2012 18:34:10 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: <226AED52-21D7-4834-A2B9-D9DDB10C4AF7@gmail.com> References: <226AED52-21D7-4834-A2B9-D9DDB10C4AF7@gmail.com> Message-ID: Hi Yury, On Tue, May 15, 2012 at 5:56 PM, Yury Selivanov wrote: > While I don't like some design quirks of Cython, I think that it's far > better than any ffi or ctypes-like solution. ?Essentially, it's an ffi > merged with the language, not a separate module. ?And that's a pretty > unique approach. ?Why not focus on improving Cython design and moving > toward some superset of Python language, that every implementation > should support natively? I'm quite sure it will never happen. See for example the new section "Pure Python Mode" in http://docs.cython.org/src/tutorial/pure.html: it is again about adding ffi-like glue to Python files. It is a reasonable next step in the Cython project, but it shows precisely that there is no chance to add all of Cython into the Python language definition. If I got you correctly, such ffi-like glue code is the part that you dislike. I agree with you, but I don't think we can hope for a solution with no glue at all; instead, we can try to minimize it as much as possible. I may be wrong, but until proven otherwise, I am under the impression that LuaJIT's approach leads to significantly less glue code than Cython's Pure Python Mode. (This is true even though the former is complete and the latter so far not: from the Cython docs, "A limited attempt is made to emulate these more complex types, but only so much can be done from the Python language.") A bient?t, Armin. From stefan_ml at behnel.de Tue May 15 18:37:08 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 May 2012 18:37:08 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: Maciej Fijalkowski, 15.05.2012 18:31: >> However, CPython doesn't have a JIT compiler, and that's one of the biggest >> drawbacks for (something like) ctypes. There's a reason we compile real >> code in Cython, not just a thin glue layer. So, if you want something >> that's usable outside of PyPy, you will have to come up with something >> that's useful in CPython as well. >> > Completely ignoring your other issues for now. No, this is not the major > drawback for ctypes for me. I know people who would happily use something > that works nicely on pypy and cpython without performance considerations. > The problem is that ctypes is at the wrong level - you cannot do some > things and that's where cython helps. Ctypes is also crazy and inconsistent > and hard to use, this makes it even more problematic. Ok, point taken. Stefan From stefan_ml at behnel.de Tue May 15 18:57:50 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 May 2012 18:57:50 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <226AED52-21D7-4834-A2B9-D9DDB10C4AF7@gmail.com> Message-ID: Armin Rigo, 15.05.2012 18:34: > On Tue, May 15, 2012 at 5:56 PM, Yury Selivanov wrote: >> While I don't like some design quirks of Cython, I think that it's far >> better than any ffi or ctypes-like solution. Essentially, it's an ffi >> merged with the language, not a separate module. And that's a pretty >> unique approach. Why not focus on improving Cython design and moving >> toward some superset of Python language, that every implementation >> should support natively? > > I'm quite sure it will never happen. See for example the new section > "Pure Python Mode" in http://docs.cython.org/src/tutorial/pure.html: > it is again about adding ffi-like glue to Python files. It is a > reasonable next step in the Cython project, but it shows precisely > that there is no chance to add all of Cython into the Python language > definition. > > If I got you correctly, such ffi-like glue code is the part that you > dislike. I agree with you, but I don't think we can hope for a > solution with no glue at all; instead, we can try to minimize it as > much as possible. I may be wrong, but until proven otherwise, I am > under the impression that LuaJIT's approach leads to significantly > less glue code than Cython's Pure Python Mode. (This is true even > though the former is complete and the latter so far not: from the > Cython docs, "A limited attempt is made to emulate these more complex > types, but only so much can be done from the Python language.") I dislike the idea of having yet another way of declaring external C APIs, though. Basically, all that's really lacking in Cython's pure Python mode is the ability to talk to C code, simply because that currently can't work from uncompiled Python code. If there was a way to build a ctypes-like thing on top of Cython's declarations, that could also be used to improve Cython's pure mode, in addition to becoming the preferred way of doing C code interaction in JIT compiled Pythons. Stefan From markflorisson88 at gmail.com Tue May 15 18:59:47 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 15 May 2012 17:59:47 +0100 Subject: [pypy-dev] Python FFI In-Reply-To: References: <226AED52-21D7-4834-A2B9-D9DDB10C4AF7@gmail.com> Message-ID: On 15 May 2012 17:34, Armin Rigo wrote: > Hi Yury, > > On Tue, May 15, 2012 at 5:56 PM, Yury Selivanov wrote: >> While I don't like some design quirks of Cython, I think that it's far >> better than any ffi or ctypes-like solution. ?Essentially, it's an ffi >> merged with the language, not a separate module. ?And that's a pretty >> unique approach. ?Why not focus on improving Cython design and moving >> toward some superset of Python language, that every implementation >> should support natively? > > I'm quite sure it will never happen. ?See for example the new section > "Pure Python Mode" in http://docs.cython.org/src/tutorial/pure.html: > it is again about adding ffi-like glue to Python files. ?It is a > reasonable next step in the Cython project, but it shows precisely > that there is no chance to add all of Cython into the Python language > definition. > > If I got you correctly, such ffi-like glue code is the part that you > dislike. ?I agree with you, but I don't think we can hope for a > solution with no glue at all; instead, we can try to minimize it as > much as possible. ?I may be wrong, but until proven otherwise, I am > under the impression that LuaJIT's approach leads to significantly > less glue code than Cython's Pure Python Mode. ?(This is true even > though the former is complete and the latter so far not: from the > Cython docs, "A limited attempt is made to emulate these more complex > types, but only so much can be done from the Python language.") > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev We have been discussing a proposal to add a mechanism to perform native calls fast given a Python object. E.g. if a JIT, or Cython code, suspects a certain signature, it can query the object for that, and if valid call the function natively through a pointer, allowing also for specializations. This could work from Cython (e.g. when calling a function with typed arguments/return value, or when casting the function object explicitly to a typed C function pointer), but this information can also be used by a JIT to call into native code quickly. The details of the proposal are here: http://wiki.cython.org/enhancements/cep1000 . I guess the question is, will this project also try to expose an interface along these lines? It would standardize an efficient mechanism to deal with wrapped native code. The CEP was not originally designed for functions wrapped by an FFI (more for callback wrappers of native code, such as numba or cython functions), but it can be used to wrap any native code. From alex.gaynor at gmail.com Tue May 15 19:21:47 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 15 May 2012 13:21:47 -0400 Subject: [pypy-dev] Python FFI In-Reply-To: References: <226AED52-21D7-4834-A2B9-D9DDB10C4AF7@gmail.com> Message-ID: On Tue, May 15, 2012 at 12:59 PM, mark florisson wrote: > On 15 May 2012 17:34, Armin Rigo wrote: > > Hi Yury, > > > > On Tue, May 15, 2012 at 5:56 PM, Yury Selivanov > wrote: > >> While I don't like some design quirks of Cython, I think that it's far > >> better than any ffi or ctypes-like solution. Essentially, it's an ffi > >> merged with the language, not a separate module. And that's a pretty > >> unique approach. Why not focus on improving Cython design and moving > >> toward some superset of Python language, that every implementation > >> should support natively? > > > > I'm quite sure it will never happen. See for example the new section > > "Pure Python Mode" in http://docs.cython.org/src/tutorial/pure.html: > > it is again about adding ffi-like glue to Python files. It is a > > reasonable next step in the Cython project, but it shows precisely > > that there is no chance to add all of Cython into the Python language > > definition. > > > > If I got you correctly, such ffi-like glue code is the part that you > > dislike. I agree with you, but I don't think we can hope for a > > solution with no glue at all; instead, we can try to minimize it as > > much as possible. I may be wrong, but until proven otherwise, I am > > under the impression that LuaJIT's approach leads to significantly > > less glue code than Cython's Pure Python Mode. (This is true even > > though the former is complete and the latter so far not: from the > > Cython docs, "A limited attempt is made to emulate these more complex > > types, but only so much can be done from the Python language.") > > > > > > A bient?t, > > > > Armin. > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > > We have been discussing a proposal to add a mechanism to perform > native calls fast given a Python object. E.g. if a JIT, or Cython > code, suspects a certain signature, it can query the object for that, > and if valid call the function natively through a pointer, allowing > also for specializations. This could work from Cython (e.g. when > calling a function with typed arguments/return value, or when casting > the function object explicitly to a typed C function pointer), but > this information can also be used by a JIT to call into native code > quickly. > > The details of the proposal are here: > http://wiki.cython.org/enhancements/cep1000 . I guess the question is, > will this project also try to expose an interface along these lines? > It would standardize an efficient mechanism to deal with wrapped > native code. The CEP was not originally designed for functions wrapped > by an FFI (more for callback wrappers of native code, such as numba or > cython functions), but it can be used to wrap any native code. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > FWIW my project is available here: https://github.com/alex/yaffi. It's very incomplete (only docs, which are also incomplete :P, ATM). Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Tue May 15 21:22:51 2012 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 15 May 2012 19:22:51 -0000 Subject: [pypy-dev] pypy-dev Digest, Vol 13, Issue 21 In-Reply-To: References: Message-ID: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> On 03:11 pm, pypy-dev-request at python.org wrote: > >Hi all, > >Fijal and me would like to raise interest among various groups of >people about building a better ctypes replacement for Python. > >The general background first, at least as far as we know it. People >generally agree that CPython extension modules are not extremely hard >to write, but also not the ultimate solution in the matter of >interfacing with C code. Obviously you need to know and write C code, >and learn the large CPython C API. Moreover a CPython extension >module is rather specific to CPython or even a particular version of >it. Even if you are lucky and your CPython extensions work on all 2.x >versions, it won't work unmodified on Python 3.x, and be slow (or not >work at all) on PyPy. (I'm leaving Jython and IronPython out of the >picture here but it's very far from ideal for them too.) > >The working alternatives nowadays are Cython and ctypes. For Cython >you have to learn a whole new language; we can argue infinitely about >preferences, but from my point of view it's mostly just a nicer way to >write CPython C extensions using --- instead of C --- a custom >language. About ctypes, I would say that the fact that its use >remains marginal is enough to show that its mission kind of failed. >Its problems are more than one, but most importantly the level at >which you describe the information is slightly too low - it's >ABI-level instead of API-level. It does not make a difference for some >libraries, but for some it does make a huge difference, like when you >use macros. We would like something that can be potentially used to >wrap almost all C libraries. > >So we would like to propose something (a bit more publically than the >two subprojects of PyPy attempting to do that --- did you know about >their existence?). The goal would be to come up with a >better-designed project. "Better designed" in my vocabulary implies >"simpler". I'm starting in this mailing list but the idea is to try >to move it in its own project, and to develop at the same time at >least a CPython 2.7 and 3.x and PyPy implementation. > >The simplest FFI we know of for a high-level language is LuaJIT's FFI. >If you are interested, read the four pages starting at >http://luajit.org/ext_ffi.html . A crucial design decision was to >avoid making use of tons of objects representing the C types; instead, >the user mostly specifies C types as plain strings, simplifying the >basic API a lot. Hi Armin, I don't think it's true that using strings instead of types (or other rich objects) simplifies anything. Quite the opposite, it takes all of the complexity which must exist and throws a huge wall up to prevent anyone from understanding it without undertaking a huge amount of extra work. C is C, whether you hide the representation of its structure in a string or whether you expose that representation as objects with useful, easy to use APIs. Jean-Paul From fijall at gmail.com Tue May 15 22:01:15 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 15 May 2012 22:01:15 +0200 Subject: [pypy-dev] pypy-dev Digest, Vol 13, Issue 21 In-Reply-To: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: On Tue, May 15, 2012 at 9:22 PM, wrote: > On 03:11 pm, pypy-dev-request at python.org wrote: > >> >> Hi all, >> >> Fijal and me would like to raise interest among various groups of >> people about building a better ctypes replacement for Python. >> >> The general background first, at least as far as we know it. People >> generally agree that CPython extension modules are not extremely hard >> to write, but also not the ultimate solution in the matter of >> interfacing with C code. Obviously you need to know and write C code, >> and learn the large CPython C API. Moreover a CPython extension >> module is rather specific to CPython or even a particular version of >> it. Even if you are lucky and your CPython extensions work on all 2.x >> versions, it won't work unmodified on Python 3.x, and be slow (or not >> work at all) on PyPy. (I'm leaving Jython and IronPython out of the >> picture here but it's very far from ideal for them too.) >> >> The working alternatives nowadays are Cython and ctypes. For Cython >> you have to learn a whole new language; we can argue infinitely about >> preferences, but from my point of view it's mostly just a nicer way to >> write CPython C extensions using --- instead of C --- a custom >> language. About ctypes, I would say that the fact that its use >> remains marginal is enough to show that its mission kind of failed. >> Its problems are more than one, but most importantly the level at >> which you describe the information is slightly too low - it's >> ABI-level instead of API-level. It does not make a difference for some >> libraries, but for some it does make a huge difference, like when you >> use macros. We would like something that can be potentially used to >> wrap almost all C libraries. >> >> So we would like to propose something (a bit more publically than the >> two subprojects of PyPy attempting to do that --- did you know about >> their existence?). The goal would be to come up with a >> better-designed project. "Better designed" in my vocabulary implies >> "simpler". I'm starting in this mailing list but the idea is to try >> to move it in its own project, and to develop at the same time at >> least a CPython 2.7 and 3.x and PyPy implementation. >> >> The simplest FFI we know of for a high-level language is LuaJIT's FFI. >> If you are interested, read the four pages starting at >> http://luajit.org/ext_ffi.html . A crucial design decision was to >> avoid making use of tons of objects representing the C types; instead, >> the user mostly specifies C types as plain strings, simplifying the >> basic API a lot. >> > > Hi Armin, > > I don't think it's true that using strings instead of types (or other rich > objects) simplifies anything. Quite the opposite, it takes all of the > complexity which must exist and throws a huge wall up to prevent anyone > from understanding it without undertaking a huge amount of extra work. > > C is C, whether you hide the representation of its structure in a string > or whether you expose that representation as objects with useful, easy to > use APIs. > > Jean-Paul > ______________________________**_________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/**mailman/listinfo/pypy-dev > The point is that using strings you can use (a subset of) C syntax. That means that you need to learn only one (and limitations) and not two ways to express stuff. I guess. Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed May 16 07:01:50 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 May 2012 07:01:50 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Maciej Fijalkowski, 15.05.2012 22:01: > On Tue, May 15, 2012 at 9:22 PM, wrote: >> I don't think it's true that using strings instead of types (or other rich >> objects) simplifies anything. Quite the opposite, it takes all of the >> complexity which must exist and throws a huge wall up to prevent anyone >> from understanding it without undertaking a huge amount of extra work. >> >> C is C, whether you hide the representation of its structure in a string >> or whether you expose that representation as objects with useful, easy to >> use APIs. > > The point is that using strings you can use (a subset of) C syntax. That > means that you need to learn only one (and limitations) and not two ways to > express stuff. I guess. I'm with Jean-Paul here. Moving code into strings just breaks editors, specifically formatting and syntax highlighting, and requires special support in IDEs in order to look up the declarations. It also blurs the line between output, documentation and pure code internals and makes code generally harder to analyse statically (not only for compilation). I don't think there's anything wrong with requiring declarations for external C code to live in external modules. It may seem like overhead for very small C APIs, but in most cases, even the initially small ones will grow large enough over time to require a suitable code modularisation on user side. Even just a struct declaration (including documentation) tends to take so much space that it's best kept out of the main source code. So why not go all the way to move declarations into importable modules all by themselves and do the auto-wrapping (or at least the necessary preprocessing) in an import hook? That allows these files to use any kind of special syntax, including plain C and C++ (with which the Cython declaration syntax is mostly identical but easier to read and parse, BTW, so nothing new to learn here). It also makes them available to code outside of a single module so that they can be reused and even shipped as packages in their own right through PyPI. Some people have already started doing that with Cython declarations, e.g. https://github.com/twiecki/CythonGSL I think there's value in that. I'm not saying that the syntax necessarily has to be exactly that of Cython. We may eventually settle on a common declaration syntax that includes special macro declarations and maybe even additional ABI information. However, I think that Cython's declaration syntax has at least largely proven its real world applicability, so something entirely new would have to stand up against that. Also note that most major IDEs already support this syntax, specifically PyCharm, PyDev and WingIDE. Stefan From arigo at tunes.org Wed May 16 08:28:25 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 16 May 2012 08:28:25 +0200 Subject: [pypy-dev] pypy-dev Digest, Vol 13, Issue 21 In-Reply-To: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi Jean-Paul, On Tue, May 15, 2012 at 9:22 PM, wrote: > I don't think it's true that using strings instead of types (or other rich > objects) simplifies anything. ?Quite the opposite, it takes all of the > complexity which must exist and throws a huge wall up to prevent anyone from > understanding it without undertaking a huge amount of extra work. While I generally agree with this, I have doubts about this particular case. It seems to me that building normal Python objects to represent C types has mostly the effect of multiplying verbosity in the source by a factor of 3, and requiring the programmer to learn details about the new classes; and with very little gains because C types are, by definition, supposed to be used only in declarations and rarely in actual code (apart from C casts). I fear that down this path lies precisely ctypes. A bient?t, Armin. From arigo at tunes.org Wed May 16 08:36:28 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 16 May 2012 08:36:28 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi Stefan, On Wed, May 16, 2012 at 7:01 AM, Stefan Behnel wrote: > I'm with Jean-Paul here. ...for a reason that might be unrelated to Jean-Paul's original complain: it's not like Cython gives first-class object manipulation on the result of parsing its .pyx files... well, too bad but Python is not Lisp :-) > I don't think there's anything wrong with requiring declarations for > external C code to live in external modules. I'm not against the idea of recognizing also Cython syntax extensions in the declarations, and indeed external files are a better idea than big inlined strings then. But I would also like to try out different paths. In one of them, these inline declarations would contain mostly just "#include " and the real header would be processed with the help of the C compiler. A bient?t, Armin. From amauryfa at gmail.com Wed May 16 09:58:00 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 16 May 2012 09:58:00 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: 2012/5/16 Armin Rigo > > I'm not against the idea of recognizing also Cython syntax extensions > in the declarations, and indeed external files are a better idea than > big inlined strings then. But I would also like to try out different > paths. In one of them, these inline declarations would contain mostly > just "#include " and the real header would be processed with > the help of the C compiler. > Another incarnation of SWIG? -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From naylor.b.david at gmail.com Wed May 16 10:07:08 2012 From: naylor.b.david at gmail.com (David Naylor) Date: Wed, 16 May 2012 10:07:08 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: <201205161007.12472.naylor.b.david@gmail.com> On Tuesday, 15 May 2012 16:23:14 Armin Rigo wrote: > Hi all, Hi, > Fijal and me would like to raise interest among various groups of > people about building a better ctypes replacement for Python. I have a suggestion, and it (possibly) involves using ctypes... > Opinions? Interests? This mail is deliberately low on details about > how we think we can do it. Instead I'm seeking general reactions for > now and would like to move this soon to its own project, independent > of PyPy. First, I think when one interfaces with a C(++) library a limited understanding of C(++) is required, at least the minimum of data types, structures and (to an extent) memory management (think pointers and who owns which structures). Using that understanding one could then use tool X to create a pythonic wrapper around the underlying C CPI and expose a "pure" python API to the rest of the world. For me, the ideal would be (using a trivial [bad] example): Notes: 1) The module "c" reads the C header file and generates (or loads a cached result, possible bundled with the program) glue that provides a low level interface to the C API described in that header file. 2) See (*) for the description of _sysctl.sysctl. The wrapper for sysctl(3) knows that the third parameter is needed by reference (and that we want the stored value), the 5th parameter None translates to NULL. 3) See (*), the wrapper knows the 5th parameter is needed by reference (and that we don't want the result). I think the trick here is automating the tedious description of the C API in ctypes (note, the above doesn't need to be done in ctypes, but can be thought of a pythonic interface to low-level C). AFAIK LLVM provides python binding for the language and (with clang) could possibly provide all the information required to parse the C header files, then python can be used to generate the interface provided by LLVM (this, to my understanding, is starting to sound like rpython...). I think it is quite possible that the interface modules for the C API could be pre-generated (in rpython, cython, etc and compiled) and distributed with the python codes (similarly to how C/Python mixed code works). Which, all the above, is very similar to the Lua FFI library, just on a different scale. Regards, David P.S. Would it not be possible for the pypy jit to inline c macros, given the right information ;-) (*) int sysctl(const int *name, u_int namelen, void *oldp, size_t *oldlenp, const void *newp, size_t newlen); -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: This is a digitally signed message part. URL: From stefan_ml at behnel.de Wed May 16 10:08:15 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 May 2012 10:08:15 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Armin Rigo, 16.05.2012 08:36: > On Wed, May 16, 2012 at 7:01 AM, Stefan Behnel wrote: >> I'm with Jean-Paul here. > > ...for a reason that might be unrelated to Jean-Paul's original > complain: it's not like Cython gives first-class object manipulation > on the result of parsing its .pyx files... I wasn't talking about .pyx files but .pxd files. While the first contain actual (Cython) source code, which the tool we are discussing here would want to see in a pure Python source file, .pxd files are meant to contain C/C++ declarations that the Cython compiler uses to learn about external code. (Also about C-level APIs of Cython generated modules, BTW, using a PyCapsule based export mechanism. We'd eventually like to remove the need for that with the CEP that Mark mentioned.) Here is an example: http://docs.cython.org/src/tutorial/clibraries.html The first page or so should give you an idea. You can stop reading at the point where I start implementing the Queue wrapper class, as that's something that you would want to do in plain Python code instead. There are ways to generate .pxd files from header files as a one-shot operation, so that users can get going quickly and then start adapting the declaration to what they want them to look like when writing their own code against them. So this indirection allows them to take advantage of some ambiguities given by C. >> I don't think there's anything wrong with requiring declarations for >> external C code to live in external modules. > > I'm not against the idea of recognizing also Cython syntax extensions > in the declarations, and indeed external files are a better idea than > big inlined strings then. But I would also like to try out different > paths. In one of them, these inline declarations would contain mostly > just "#include " and the real header would be processed with > the help of the C compiler. Then take a look at some of the tools that Cython users have written to generate .pxd files from header files. I heard of at least two that use gccxml and clang respectively. Seems like Wim Lavrijsen has also started doing the same thing all over again with cppyy. SWIG also does it, but basically all on its own and without the helpful hand of a real C/C++ compiler. I would really appreciate it if there was one way to do this that everyone could just settle on and use, instead of throwing massive amounts of separate developer time on the same issue again and again, that only lead to tools that "work for what the author needed right now". We actually have a GSoC project this year that tries to get a .pxd generation tool up to a point where it becomes generally usable, based on the gcc Python plugin, which sounds like a good idea to me. There's still some time left to refocus this into something that also makes it more widely usable, outside of a plain Cython context. Stefan From arigo at tunes.org Wed May 16 10:09:03 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 16 May 2012 10:09:03 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi Amaury, On Wed, May 16, 2012 at 9:58 AM, Amaury Forgeot d'Arc wrote: > Another incarnation of SWIG? ...I suppose you can say so a priori. The difference is that we won't try to perform conversions for the user. Instead he gets his instances of C types as real Python objects and, if needed, needs to write conversions directly in Python. A bient?t, Armin. From estama at gmail.com Wed May 16 12:54:32 2012 From: estama at gmail.com (Eleytherios Stamatogiannakis) Date: Wed, 16 May 2012 13:54:32 +0300 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: <4FB38768.5040405@gmail.com> Is there in the plans a Jit friendly callback mechanism (from C to pypy) for the new API? IMHO, right now callback performance in pypy is very bad (2x to 12x slower then CPython), and there is no way to do it efficiently apart from going the RPython way and retranslating all of pypy. l. From arigo at tunes.org Wed May 16 14:47:02 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 16 May 2012 14:47:02 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: <4FB38768.5040405@gmail.com> References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> <4FB38768.5040405@gmail.com> Message-ID: Hi, On Wed, May 16, 2012 at 12:54 PM, Eleytherios Stamatogiannakis wrote: > Is there in the plans a Jit friendly callback mechanism (from C to pypy) for > the new API? It is far too early to have such plans, but I guess so. Armin From wlavrijsen at lbl.gov Wed May 16 17:39:14 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 16 May 2012 08:39:14 -0700 (PDT) Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi Stefan, > Then take a look at some of the tools that Cython users have written to > generate .pxd files from header files. I heard of at least two that use > gccxml and clang respectively. Seems like Wim Lavrijsen has also started > doing the same thing all over again with cppyy. not sure where the statement "all over again" comes from, given that the tech we use for CPython-bindings pre-dates Cython by half a decade or more. Now it's just being re-used and re-implemented for PyPy in a way that fits PyPy best. > I would really appreciate it if there was one way to do this that everyone > could just settle on and use, instead of throwing massive amounts of > separate developer time on the same issue again and again, that only lead > to tools that "work for what the author needed right now". That'd be great, but we (HEP) absolutely need C++ in all it's ugly glory, and it needs to be fully automatic from the header files. W/o that ability, which we do have today for CPython, there would be no Python in HEP. This discussion seems to be mostly restricted to C. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From bokr at oz.net Wed May 16 17:54:47 2012 From: bokr at oz.net (Bengt Richter) Date: Wed, 16 May 2012 08:54:47 -0700 Subject: [pypy-dev] pypy -mime rfc2045_file? -- Was/is?: Re: Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: <4FB3CDC7.8050708@oz.net> Hi all, I hope I did the right thing changing the subject line, though it was the ffi C header and CPython mixed source issues that got me started... On 05/15/2012 11:36 PM Armin Rigo wrote: > Hi Stefan, > > On Wed, May 16, 2012 at 7:01 AM, Stefan Behnel wrote: >> I'm with Jean-Paul here. > > ...for a reason that might be unrelated to Jean-Paul's original > complain: it's not like Cython gives first-class object manipulation > on the result of parsing its .pyx files... well, too bad but Python > is not Lisp :-) > >> I don't think there's anything wrong with requiring declarations for >> external C code to live in external modules. > > I'm not against the idea of recognizing also Cython syntax extensions > in the declarations, and indeed external files are a better idea than > big inlined strings then. But I would also like to try out different > paths. In one of them, these inline declarations would contain mostly > just "#include" and the real header would be processed with > the help of the C compiler. > > > A bient?t, > > Armin. I see in this a general method of passing arbitrary sources to a translation system -- in this case with pypy as the invoking shell command. with now a possibly open-ended selection of file types. After C headers, CPython, ... ?? ;-) rfc2045[1] and related rfcs define the MIME way of specifying multipart data sets, with open-ended ways of specifying types, character encoding, transmission encoding, etc., and includes a simple delimiter system that permits nested quoting/inclusion implying tree structure possibilities (unlikely, but if needed) without too much judo. [1] http://tools.ietf.org/html/rfc2045 So I'm wondering if you could see using a new command line option something like pypy -mime mime_container_file -mime optionally_another python_source You could even optionally package everything including python sources in the container file, specifying what's what in the mime headers. Also wouldn't need encoding cookies in the source bodies. Or you could just use the mime file as a top level type and encoding specification for the rest of your sources, like a typed manifest for what your pypy invocation is going to use. I am not totally in love with MIME, but I like the abstraction behind it. And it is a pretty well defined standard in common use (which solves a big part of the documentation problem ;-) I would think that using a general-purpose multipart container file type as (optional) source would help with IDEs also, making it easy to make a folded top level MIME representation that can be expanded by clicking a [+] line to invoke plugin specialized editors selected on the basis of the metadata headers in the MIME. (I guess you could do the same with any container file, .e.g. .tgz, but I suspect you'd wind up re-inventing conventions already defined in rfc2045). Further, there's no reason you couldn't define MIME parts that were just a line or two of headers, a blank separator, and some url lines pointing to files anywhere. You could include sha1 hashes or gpg signatures etc in the header section for automated security and integrity checks, specify caching and/or automatic retrieval or update, etc. etc. But you could also use MIME very minimally ;-) This would be a way to include read-only legacy files with no modifications, while specifying encoding etc in a mixed type set that would otherwise require encoding cookie mods. You could also include or refer to patch or test stuff, but you could also use single-part MIME files like typed symbolic links with only a single line of metadata, a blank line, and an url in the file. I'm sure you are already ahead of me into other possibilities. Regards, Bengt Richter From wlavrijsen at lbl.gov Wed May 16 17:56:56 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 16 May 2012 08:56:56 -0700 (PDT) Subject: [pypy-dev] Python FFI In-Reply-To: <201205161007.12472.naylor.b.david@gmail.com> References: <201205161007.12472.naylor.b.david@gmail.com> Message-ID: Hi David, > I think the trick here is automating the tedious description of the C API in > ctypes (note, the above doesn't need to be done in ctypes, but can be thought > of a pythonic interface to low-level C). AFAIK LLVM provides python binding > for the language and (with clang) could possibly provide all the information > required to parse the C header files, then python can be used to generate the > interface provided by LLVM (this, to my understanding, is starting to sound > like rpython...). > > I think it is quite possible that the interface modules for the C API could be > pre-generated (in rpython, cython, etc and compiled) and distributed with the > python codes (similarly to how C/Python mixed code works). years ago, I did this once for ctypes, as a proof. I used Reflex to generate the reflection info from the Python.h header, then Relax and PyROOT to extract what was needed for decorating libpython.so in Python, then put it into a dict and wrote that out into a pickle file. With the tools in place (the above is quite a dependency set if they aren't yet), that only took 150 lines of Python or so (which could be used on any header, to be sure, not just Python.h) and the pickle file was all that was needed beyond that to decorate the libpython.so library for ctypes with for each subsequent run. (You'd need one pickle file per platform and python version (i.e. header) combination in this setup.) Using CLang would work just as well (except that it's still not a very stable API for now, but that will happen). Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From stefan_ml at behnel.de Wed May 16 18:25:28 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 May 2012 18:25:28 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: wlavrijsen at lbl.gov, 16.05.2012 17:39: >> Then take a look at some of the tools that Cython users have written to >> generate .pxd files from header files. I heard of at least two that use >> gccxml and clang respectively. Seems like Wim Lavrijsen has also started >> doing the same thing all over again with cppyy. > > not sure where the statement "all over again" comes from, given that the > tech we use for CPython-bindings pre-dates Cython by half a decade or more. Hmm, are you sure? Pyrex is pretty old, too. Cython's C++ support certainly is a lot younger, though. Well, anyway... My point is that there are too many ways to extract the declarations of a C/C++ API already and each of them is tied to a specific tool and thus hard to reuse elsewhere. It would be great to have a common declaration layer that these tools can generate from the one side and code can use from the other side. > Now it's just being re-used and re-implemented for PyPy in a way that fits > PyPy best. Ok, my bad then. Get in line with SWIG. ;) >> I would really appreciate it if there was one way to do this that everyone >> could just settle on and use, instead of throwing massive amounts of >> separate developer time on the same issue again and again, that only lead >> to tools that "work for what the author needed right now". > > That'd be great, but we (HEP) absolutely need C++ in all it's ugly glory, > and it needs to be fully automatic from the header files. W/o that ability, > which we do have today for CPython, there would be no Python in HEP. This > discussion seems to be mostly restricted to C. I admit that I might have become a bit sloppy about all this "C/C++" repetition over time. I definitely see C++ in the loop here. Stefan From wlavrijsen at lbl.gov Wed May 16 19:32:07 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 16 May 2012 10:32:07 -0700 (PDT) Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi Stefan, > Hmm, are you sure? Pyrex is pretty old, too. Cython's C++ support certainly > is a lot younger, though. Well, anyway... we've been parsing C++ since C++ itself was a spring chicken. I don't know when Pyrex started (and it doesn't seem to parse headers AFAICS?), but unless you tell me it was the early nineties ... The Python bindings came later for us (2002), and again, no other technology would have flown: we simply would not have been using Python today in HEP if I'd had followed any other route. > My point is that there are too many ways to extract the declarations of a > C/C++ API already and each of them is tied to a specific tool and thus hard > to reuse elsewhere. This is true and most tools are completely insufficient for all tasks as C++ parsing is hard, so everyone always settles on a subset. CLang finally changes this picture, which is why we are betting on that. Problem is inertia, and the fact that the reflection info is also used for I/O (that was its original purpose). Transitioning is painful, as not everyone here is convinced that allowing the tech to read our data be an external tool is acceptable, given that we're thinking on timescales of the order of decades, and you never know when anything not in-house goes dodo-bird or is taken private. >> Now it's just being re-used and re-implemented for PyPy in a way that fits >> PyPy best. > > Ok, my bad then. Get in line with SWIG. ;) Not sure I'm following that one: back in the day, we tried SWIG, but that wasn't up to the task (it's much better today, but not yet good enough). The point I'm making is that I could get the original CPython extension code to work on PyPy. However, then the Python side is fast (b/c of the JIT), and the C++ side is fast (b/c it's C++), leaving the bindings to become the bottle neck and we'd have achieved very little. That same issue is true for SWIG. With cppyy, which uses libffi underneath and thus readily lifts on the work done by the core PyPy folks for ctypes, virtually all call overhead can be removed. Only overhead left are a couple of guards. However, if the C++ body isn't completely trivial, then OOO and hyperthreading happily munch those guards on the side. Iow. on a modern CPU that isn't completely overcommitted, the cost of calling from Python into C++ can be as low as the cost of calling from C++ into C++. (And it's only a fraction more if overcommitted.) That's the point of cppyy, not the parsing/extracting part. The parsing part is always an external tool. Whether Reflex, CINT, Cling, or Cython++ when that becomes available for use, makes no matter. Build it (to the same level that any of those other tools are today), and I'll add a back-end. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From needham.f at gmail.com Thu May 17 23:18:31 2012 From: needham.f at gmail.com (Needham, Fanny) Date: Thu, 17 May 2012 22:18:31 +0100 Subject: [pypy-dev] placement Message-ID: <07423105.20120517221831@gmail.com> Dear Pypy-dev, How is your web site doing? Email us today for a no-cost web performance report and a no-hassle consultation. Sincerely, Fanny Needham Interplay-Marketing pypy-dev at codespeak.net 5/17/2012 From sasikanth.ece at gmail.com Fri May 18 06:58:11 2012 From: sasikanth.ece at gmail.com (Sasikanth Eda) Date: Fri, 18 May 2012 10:28:11 +0530 Subject: [pypy-dev] PyPy proved slower than Python when used os.rename and os.link [ Suggest how to get rid of this ] Message-ID: Hai All, In trails of finding the execution time lapse between Python and PyPy , its proved that PyPy is slower than Python. Here are the trials done ; *Step-1: Written a Python code that uses repeated os.rename()* * * *code: (file name : rename.py)* * #!/usr/bin/env python* * * * import os* * import sys* * * * os.chdir("/opt/pypy-1.8/bin")* * print "Rename count -1 (a1 -> a2)"* * os.rename("a1","a2")* * print "Rename count -2 (a2 -> a3)"* * os.rename("a2","a3")* * print "Rename count -3 (a3 -> a4)"* * os.rename("a3","a4")* * print "Rename count -4 (a4 -> a5)"* * os.rename("a4","a5")* * print "Rename count -5 (a5 -> a6)"* * os.rename("a5","a6")* * print "Rename count -6 (a6 -> a7)"* * os.rename("a6","a7")* * print "Rename count -7 (a7 -> a8)"* * os.rename("a7","a8")* * print "Rename count -8 (a8 -> a9)"* * os.rename("a8","a9")* * print "Rename count -9 (a9 -> a0)"* * os.rename("a9","a0")* * print "Rename count -10 (a0 -> B0)"* * os.rename("a0","B0")* * * *Step-2: Observed Execution time with Python 2.7* * * * [root at Manojkiran bin]# time python rename.py * * Rename count -1 (a1 -> a2)* * Rename count -2 (a2 -> a3)* * Rename count -3 (a3 -> a4)* * Rename count -4 (a4 -> a5)* * Rename count -5 (a5 -> a6)* * Rename count -6 (a6 -> a7)* * Rename count -7 (a7 -> a8)* * Rename count -8 (a8 -> a9)* * Rename count -9 (a9 -> a0)* * Rename count -10 (a0 -> B0)* * * * real 0m0.031s* * user 0m0.021s* * sys 0m0.010s* Step-3:* Observed Execution time with PyPy 1.8* * * [root at Manojkiran bin]# time ./pypy rename.py * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version information available (required by ./pypy)* * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version information available (required by ./pypy)* * Rename count -1 (a1 -> a2)* * Rename count -2 (a2 -> a3)* * Rename count -3 (a3 -> a4)* * Rename count -4 (a4 -> a5)* * Rename count -5 (a5 -> a6)* * Rename count -6 (a6 -> a7)* * Rename count -7 (a7 -> a8)* * Rename count -8 (a8 -> a9)* * Rename count -9 (a9 -> a0)* * Rename count -10 (a0 -> B0)* real 0m0.054s user 0m0.036s sys 0m0.016s * * *Step-4: Written a Python code that uses repeated os.link()* * * *code: (file name : link.py)* * #!/usr/bin/env python* * * * import os* * import sys* * * * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l0")* * print "Linked count -1 (l0 -> lo)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l1")* * print "Linked count -2 (l0 -> l1)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l2")* * print "Linked count -3 (l0 -> l2)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l3")* * print "Linked count -4 (l0 -> l3)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l4")* * print "Linked count -5 (l0 -> l4)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l5")* * print "Linked count -6 (l0 -> l5)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l6")* * print "Linked count -7 (l0 -> l6)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l7")* * print "Linked count -8 (l0 -> l7)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l8")* * print "Linked count -9 (l0 -> l8)"* * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l9")* * print "Linked count -10 (l0 -> l9)"* *Step-5: Observed Execution time with Python 2.7* * * * [root at Manojkiran bin]# time python link.py * * Linked count -1 (l0 -> l0)* * **Linked** count -2 (l0 -> l1)* * **Linked** count -3 (l0 -> l2)* * **Linked** count -4 (l0 -> l3)* * **Linked** count -5 (l0 -> l4)* * **Linked** count -6 (l0 -> l5)* * **Linked** count -7 (l0 -> l6)* * **Linked** count -8 (l0 -> l7)* * **Linked** count -9 (l0 -> l8)* * **Linked** count -10 (l0 -> l9)* * * * real 0m0.028s* * user 0m0.020s* * sys 0m0.008s* Step-6:* Observed Execution time with PyPy 1.8* * * [root at Manojkiran bin]# time ./pypy link.py * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version information available (required by ./pypy)* * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version information available (required by ./pypy)* * Linked count -1 (l0 -> l0)* * **Linked** count -2 (l0 -> l1)* * **Linked** count -3 (l0 -> l2)* * **Linked** count -4 (l0 -> l3)* * **Linked** count -5 (l0 -> l4)* * **Linked** count -6 (l0 -> l5)* * **Linked** count -7 (l0 -> l6)* * **Linked** count -8 (l0 -> l7)* * **Linked** count -9 (l0 -> l8)* * **Linked** count -10 (l0 -> l9)* * * real 0m0.056s user 0m0.032s sys 0m0.023s Hence in my understanding Python is better in terms of execution time when compared with PyPy. Kindly suggest me if my trails are proper and why PyPy failed to achieve better speed then PyPy ? Is this a know issue or whether we have any fix for this ? Thanking you, -- Sasikanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Fri May 18 07:11:21 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 18 May 2012 01:11:21 -0400 Subject: [pypy-dev] PyPy proved slower than Python when used os.rename and os.link [ Suggest how to get rid of this ] In-Reply-To: References: Message-ID: On Fri, May 18, 2012 at 12:58 AM, Sasikanth Eda wrote: > Hai All, > > In trails of finding the execution time lapse between Python and PyPy , > its proved that PyPy is slower than Python. > > Here are the trials done ; > > *Step-1: Written a Python code that uses repeated os.rename()* > * > * > *code: (file name : rename.py)* > * #!/usr/bin/env python* > * > * > * import os* > * import sys* > * > * > * os.chdir("/opt/pypy-1.8/bin")* > * print "Rename count -1 (a1 -> a2)"* > * os.rename("a1","a2")* > * print "Rename count -2 (a2 -> a3)"* > * os.rename("a2","a3")* > * print "Rename count -3 (a3 -> a4)"* > * os.rename("a3","a4")* > * print "Rename count -4 (a4 -> a5)"* > * os.rename("a4","a5")* > * print "Rename count -5 (a5 -> a6)"* > * os.rename("a5","a6")* > * print "Rename count -6 (a6 -> a7)"* > * os.rename("a6","a7")* > * print "Rename count -7 (a7 -> a8)"* > * os.rename("a7","a8")* > * print "Rename count -8 (a8 -> a9)"* > * os.rename("a8","a9")* > * print "Rename count -9 (a9 -> a0)"* > * os.rename("a9","a0")* > * print "Rename count -10 (a0 -> B0)"* > * os.rename("a0","B0")* > * > * > *Step-2: Observed Execution time with Python 2.7* > * > * > * [root at Manojkiran bin]# time python rename.py * > * Rename count -1 (a1 -> a2)* > * Rename count -2 (a2 -> a3)* > * Rename count -3 (a3 -> a4)* > * Rename count -4 (a4 -> a5)* > * Rename count -5 (a5 -> a6)* > * Rename count -6 (a6 -> a7)* > * Rename count -7 (a7 -> a8)* > * Rename count -8 (a8 -> a9)* > * Rename count -9 (a9 -> a0)* > * Rename count -10 (a0 -> B0)* > * > * > * real 0m0.031s* > * user 0m0.021s* > * sys 0m0.010s* > > Step-3:* Observed Execution time with PyPy 1.8* > * > * > [root at Manojkiran bin]# time ./pypy rename.py > * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version > information available (required by ./pypy)* > * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version > information available (required by ./pypy)* > * Rename count -1 (a1 -> a2)* > * Rename count -2 (a2 -> a3)* > * Rename count -3 (a3 -> a4)* > * Rename count -4 (a4 -> a5)* > * Rename count -5 (a5 -> a6)* > * Rename count -6 (a6 -> a7)* > * Rename count -7 (a7 -> a8)* > * Rename count -8 (a8 -> a9)* > * Rename count -9 (a9 -> a0)* > * Rename count -10 (a0 -> B0)* > > real 0m0.054s > user 0m0.036s > sys 0m0.016s > * > * > *Step-4: Written a Python code that uses repeated os.link()* > * > * > *code: (file name : link.py)* > * #!/usr/bin/env python* > * > * > * import os* > * import sys* > * > * > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l0")* > * print "Linked count -1 (l0 -> lo)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l1")* > * print "Linked count -2 (l0 -> l1)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l2")* > * print "Linked count -3 (l0 -> l2)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l3")* > * print "Linked count -4 (l0 -> l3)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l4")* > * print "Linked count -5 (l0 -> l4)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l5")* > * print "Linked count -6 (l0 -> l5)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l6")* > * print "Linked count -7 (l0 -> l6)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l7")* > * print "Linked count -8 (l0 -> l7)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l8")* > * print "Linked count -9 (l0 -> l8)"* > * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l9")* > * print "Linked count -10 (l0 -> l9)"* > > *Step-5: Observed Execution time with Python 2.7* > * > * > * [root at Manojkiran bin]# time python link.py * > * Linked count -1 (l0 -> l0)* > * **Linked** count -2 (l0 -> l1)* > * **Linked** count -3 (l0 -> l2)* > * **Linked** count -4 (l0 -> l3)* > * **Linked** count -5 (l0 -> l4)* > * **Linked** count -6 (l0 -> l5)* > * **Linked** count -7 (l0 -> l6)* > * **Linked** count -8 (l0 -> l7)* > * **Linked** count -9 (l0 -> l8)* > * **Linked** count -10 (l0 -> l9)* > * > * > * real 0m0.028s* > * user 0m0.020s* > * sys 0m0.008s* > > Step-6:* Observed Execution time with PyPy 1.8* > * > * > [root at Manojkiran bin]# time ./pypy link.py > * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version > information available (required by ./pypy)* > * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version > information available (required by ./pypy)* > * Linked count -1 (l0 -> l0)* > * **Linked** count -2 (l0 -> l1)* > * **Linked** count -3 (l0 -> l2)* > * **Linked** count -4 (l0 -> l3)* > * **Linked** count -5 (l0 -> l4)* > * **Linked** count -6 (l0 -> l5)* > * **Linked** count -7 (l0 -> l6)* > * **Linked** count -8 (l0 -> l7)* > * **Linked** count -9 (l0 -> l8)* > * **Linked** count -10 (l0 -> l9)* > * > * > real 0m0.056s > user 0m0.032s > sys 0m0.023s > > Hence in my understanding Python is better in terms of execution time when > compared with PyPy. > > Kindly suggest me if my trails are proper and why PyPy failed to achieve > better speed then PyPy ? > > Is this a know issue or whether we have any fix for this ? > > Thanking you, > -- > Sasikanth > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > PyPy's JIT works by optimizing loops and functions that are frequently run, this generally means your code needs to run at least .5 seconds for the JIT to kick in and show benefits. Your code contains no loops for the JIT to optimize. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From sasikanth.ece at gmail.com Fri May 18 07:42:44 2012 From: sasikanth.ece at gmail.com (Sasikanth Eda) Date: Fri, 18 May 2012 11:12:44 +0530 Subject: [pypy-dev] PyPy proved slower than Python when used os.rename and os.link [ Suggest how to get rid of this ] In-Reply-To: References: Message-ID: Hai Alex, Thanks for your reply and making me understand about PyPy Jit. Some more query on PyPy; 1. In my trails I agree that I don't have loops in my program so PyPy is not able to optimize it. But it should at least should give me the same execution time as when executed with python. So what might be the reason for this that PyPy took more time than Python ? 2. Suppose assume a condition where we have a lot of loops and function calls in a python program and it is called by several independent processes (a typical cgi-model). Then what could be the thought is we use PyPy for such programs (i.e in specific whether are we going to get any performance changes when compared with python ? ) Thanking you, Sasikanth On Fri, May 18, 2012 at 10:41 AM, Alex Gaynor wrote: > > > On Fri, May 18, 2012 at 12:58 AM, Sasikanth Eda wrote: > >> Hai All, >> >> In trails of finding the execution time lapse between Python and PyPy , >> its proved that PyPy is slower than Python. >> >> Here are the trials done ; >> >> *Step-1: Written a Python code that uses repeated os.rename()* >> * >> * >> *code: (file name : rename.py)* >> * #!/usr/bin/env python* >> * >> * >> * import os* >> * import sys* >> * >> * >> * os.chdir("/opt/pypy-1.8/bin")* >> * print "Rename count -1 (a1 -> a2)"* >> * os.rename("a1","a2")* >> * print "Rename count -2 (a2 -> a3)"* >> * os.rename("a2","a3")* >> * print "Rename count -3 (a3 -> a4)"* >> * os.rename("a3","a4")* >> * print "Rename count -4 (a4 -> a5)"* >> * os.rename("a4","a5")* >> * print "Rename count -5 (a5 -> a6)"* >> * os.rename("a5","a6")* >> * print "Rename count -6 (a6 -> a7)"* >> * os.rename("a6","a7")* >> * print "Rename count -7 (a7 -> a8)"* >> * os.rename("a7","a8")* >> * print "Rename count -8 (a8 -> a9)"* >> * os.rename("a8","a9")* >> * print "Rename count -9 (a9 -> a0)"* >> * os.rename("a9","a0")* >> * print "Rename count -10 (a0 -> B0)"* >> * os.rename("a0","B0")* >> * >> * >> *Step-2: Observed Execution time with Python 2.7* >> * >> * >> * [root at Manojkiran bin]# time python rename.py * >> * Rename count -1 (a1 -> a2)* >> * Rename count -2 (a2 -> a3)* >> * Rename count -3 (a3 -> a4)* >> * Rename count -4 (a4 -> a5)* >> * Rename count -5 (a5 -> a6)* >> * Rename count -6 (a6 -> a7)* >> * Rename count -7 (a7 -> a8)* >> * Rename count -8 (a8 -> a9)* >> * Rename count -9 (a9 -> a0)* >> * Rename count -10 (a0 -> B0)* >> * >> * >> * real 0m0.031s* >> * user 0m0.021s* >> * sys 0m0.010s* >> >> Step-3:* Observed Execution time with PyPy 1.8* >> * >> * >> [root at Manojkiran bin]# time ./pypy rename.py >> * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version >> information available (required by ./pypy)* >> * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version >> information available (required by ./pypy)* >> * Rename count -1 (a1 -> a2)* >> * Rename count -2 (a2 -> a3)* >> * Rename count -3 (a3 -> a4)* >> * Rename count -4 (a4 -> a5)* >> * Rename count -5 (a5 -> a6)* >> * Rename count -6 (a6 -> a7)* >> * Rename count -7 (a7 -> a8)* >> * Rename count -8 (a8 -> a9)* >> * Rename count -9 (a9 -> a0)* >> * Rename count -10 (a0 -> B0)* >> >> real 0m0.054s >> user 0m0.036s >> sys 0m0.016s >> * >> * >> *Step-4: Written a Python code that uses repeated os.link()* >> * >> * >> *code: (file name : link.py)* >> * #!/usr/bin/env python* >> * >> * >> * import os* >> * import sys* >> * >> * >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l0")* >> * print "Linked count -1 (l0 -> lo)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l1")* >> * print "Linked count -2 (l0 -> l1)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l2")* >> * print "Linked count -3 (l0 -> l2)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l3")* >> * print "Linked count -4 (l0 -> l3)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l4")* >> * print "Linked count -5 (l0 -> l4)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l5")* >> * print "Linked count -6 (l0 -> l5)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l6")* >> * print "Linked count -7 (l0 -> l6)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l7")* >> * print "Linked count -8 (l0 -> l7)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l8")* >> * print "Linked count -9 (l0 -> l8)"* >> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l9")* >> * print "Linked count -10 (l0 -> l9)"* >> >> *Step-5: Observed Execution time with Python 2.7* >> * >> * >> * [root at Manojkiran bin]# time python link.py * >> * Linked count -1 (l0 -> l0)* >> * **Linked** count -2 (l0 -> l1)* >> * **Linked** count -3 (l0 -> l2)* >> * **Linked** count -4 (l0 -> l3)* >> * **Linked** count -5 (l0 -> l4)* >> * **Linked** count -6 (l0 -> l5)* >> * **Linked** count -7 (l0 -> l6)* >> * **Linked** count -8 (l0 -> l7)* >> * **Linked** count -9 (l0 -> l8)* >> * **Linked** count -10 (l0 -> l9)* >> * >> * >> * real 0m0.028s* >> * user 0m0.020s* >> * sys 0m0.008s* >> >> Step-6:* Observed Execution time with PyPy 1.8* >> * >> * >> [root at Manojkiran bin]# time ./pypy link.py >> * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version >> information available (required by ./pypy)* >> * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version >> information available (required by ./pypy)* >> * Linked count -1 (l0 -> l0)* >> * **Linked** count -2 (l0 -> l1)* >> * **Linked** count -3 (l0 -> l2)* >> * **Linked** count -4 (l0 -> l3)* >> * **Linked** count -5 (l0 -> l4)* >> * **Linked** count -6 (l0 -> l5)* >> * **Linked** count -7 (l0 -> l6)* >> * **Linked** count -8 (l0 -> l7)* >> * **Linked** count -9 (l0 -> l8)* >> * **Linked** count -10 (l0 -> l9)* >> * >> * >> real 0m0.056s >> user 0m0.032s >> sys 0m0.023s >> >> Hence in my understanding Python is better in terms of execution time >> when compared with PyPy. >> >> Kindly suggest me if my trails are proper and why PyPy failed to achieve >> better speed then PyPy ? >> >> Is this a know issue or whether we have any fix for this ? >> >> Thanking you, >> -- >> Sasikanth >> >> >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> > PyPy's JIT works by optimizing loops and functions that are frequently > run, this generally means your code needs to run at least .5 seconds for > the JIT to kick in and show benefits. Your code contains no loops for the > JIT to optimize. > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Fri May 18 07:46:30 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 18 May 2012 01:46:30 -0400 Subject: [pypy-dev] PyPy proved slower than Python when used os.rename and os.link [ Suggest how to get rid of this ] In-Reply-To: References: Message-ID: On Fri, May 18, 2012 at 1:42 AM, Sasikanth Eda wrote: > Hai Alex, > > Thanks for your reply and making me understand about PyPy Jit. > > Some more query on PyPy; > > 1. In my trails I agree that I don't have loops in my program so PyPy is > not able to optimize it. But it should at least should give me the same > execution time as when executed with python. > So what might be the reason for this that PyPy took more time than > Python ? > > PyPy's interpreter isn't as fast as CPython, it's only with our JIT that we are faster for most code. > 2. Suppose assume a condition where we have a lot of loops and function > calls in a python program and it is called by several independent processes > (a typical cgi-model). Then what could be the thought is we use PyPy for > such programs (i.e in specific whether are we going to get any performance > changes when compared with python ? ) > > This is a very non-optimal case for PyPy, a persistant server model, as used by most WSGI servers would be much better. Alex > Thanking you, > Sasikanth > > On Fri, May 18, 2012 at 10:41 AM, Alex Gaynor wrote: > >> >> >> On Fri, May 18, 2012 at 12:58 AM, Sasikanth Eda wrote: >> >>> Hai All, >>> >>> In trails of finding the execution time lapse between Python and PyPy , >>> its proved that PyPy is slower than Python. >>> >>> Here are the trials done ; >>> >>> *Step-1: Written a Python code that uses repeated os.rename()* >>> * >>> * >>> *code: (file name : rename.py)* >>> * #!/usr/bin/env python* >>> * >>> * >>> * import os* >>> * import sys* >>> * >>> * >>> * os.chdir("/opt/pypy-1.8/bin")* >>> * print "Rename count -1 (a1 -> a2)"* >>> * os.rename("a1","a2")* >>> * print "Rename count -2 (a2 -> a3)"* >>> * os.rename("a2","a3")* >>> * print "Rename count -3 (a3 -> a4)"* >>> * os.rename("a3","a4")* >>> * print "Rename count -4 (a4 -> a5)"* >>> * os.rename("a4","a5")* >>> * print "Rename count -5 (a5 -> a6)"* >>> * os.rename("a5","a6")* >>> * print "Rename count -6 (a6 -> a7)"* >>> * os.rename("a6","a7")* >>> * print "Rename count -7 (a7 -> a8)"* >>> * os.rename("a7","a8")* >>> * print "Rename count -8 (a8 -> a9)"* >>> * os.rename("a8","a9")* >>> * print "Rename count -9 (a9 -> a0)"* >>> * os.rename("a9","a0")* >>> * print "Rename count -10 (a0 -> B0)"* >>> * os.rename("a0","B0")* >>> * >>> * >>> *Step-2: Observed Execution time with Python 2.7* >>> * >>> * >>> * [root at Manojkiran bin]# time python rename.py * >>> * Rename count -1 (a1 -> a2)* >>> * Rename count -2 (a2 -> a3)* >>> * Rename count -3 (a3 -> a4)* >>> * Rename count -4 (a4 -> a5)* >>> * Rename count -5 (a5 -> a6)* >>> * Rename count -6 (a6 -> a7)* >>> * Rename count -7 (a7 -> a8)* >>> * Rename count -8 (a8 -> a9)* >>> * Rename count -9 (a9 -> a0)* >>> * Rename count -10 (a0 -> B0)* >>> * >>> * >>> * real 0m0.031s* >>> * user 0m0.021s* >>> * sys 0m0.010s* >>> >>> Step-3:* Observed Execution time with PyPy 1.8* >>> * >>> * >>> [root at Manojkiran bin]# time ./pypy rename.py >>> * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version >>> information available (required by ./pypy)* >>> * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version >>> information available (required by ./pypy)* >>> * Rename count -1 (a1 -> a2)* >>> * Rename count -2 (a2 -> a3)* >>> * Rename count -3 (a3 -> a4)* >>> * Rename count -4 (a4 -> a5)* >>> * Rename count -5 (a5 -> a6)* >>> * Rename count -6 (a6 -> a7)* >>> * Rename count -7 (a7 -> a8)* >>> * Rename count -8 (a8 -> a9)* >>> * Rename count -9 (a9 -> a0)* >>> * Rename count -10 (a0 -> B0)* >>> >>> real 0m0.054s >>> user 0m0.036s >>> sys 0m0.016s >>> * >>> * >>> *Step-4: Written a Python code that uses repeated os.link()* >>> * >>> * >>> *code: (file name : link.py)* >>> * #!/usr/bin/env python* >>> * >>> * >>> * import os* >>> * import sys* >>> * >>> * >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l0")* >>> * print "Linked count -1 (l0 -> lo)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l1")* >>> * print "Linked count -2 (l0 -> l1)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l2")* >>> * print "Linked count -3 (l0 -> l2)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l3")* >>> * print "Linked count -4 (l0 -> l3)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l4")* >>> * print "Linked count -5 (l0 -> l4)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l5")* >>> * print "Linked count -6 (l0 -> l5)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l6")* >>> * print "Linked count -7 (l0 -> l6)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l7")* >>> * print "Linked count -8 (l0 -> l7)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l8")* >>> * print "Linked count -9 (l0 -> l8)"* >>> * os.link("/opt/pypy-1.8/bin/l0","/opt/pypy-1.8/bin/test/l9")* >>> * print "Linked count -10 (l0 -> l9)"* >>> >>> *Step-5: Observed Execution time with Python 2.7* >>> * >>> * >>> * [root at Manojkiran bin]# time python link.py * >>> * Linked count -1 (l0 -> l0)* >>> * **Linked** count -2 (l0 -> l1)* >>> * **Linked** count -3 (l0 -> l2)* >>> * **Linked** count -4 (l0 -> l3)* >>> * **Linked** count -5 (l0 -> l4)* >>> * **Linked** count -6 (l0 -> l5)* >>> * **Linked** count -7 (l0 -> l6)* >>> * **Linked** count -8 (l0 -> l7)* >>> * **Linked** count -9 (l0 -> l8)* >>> * **Linked** count -10 (l0 -> l9)* >>> * >>> * >>> * real 0m0.028s* >>> * user 0m0.020s* >>> * sys 0m0.008s* >>> >>> Step-6:* Observed Execution time with PyPy 1.8* >>> * >>> * >>> [root at Manojkiran bin]# time ./pypy link.py >>> * ./pypy: /usr/local/ssl/lib/libssl.so.0.9.8: no version >>> information available (required by ./pypy)* >>> * ./pypy: /usr/local/ssl/lib/libcrypto.so.0.9.8: no version >>> information available (required by ./pypy)* >>> * Linked count -1 (l0 -> l0)* >>> * **Linked** count -2 (l0 -> l1)* >>> * **Linked** count -3 (l0 -> l2)* >>> * **Linked** count -4 (l0 -> l3)* >>> * **Linked** count -5 (l0 -> l4)* >>> * **Linked** count -6 (l0 -> l5)* >>> * **Linked** count -7 (l0 -> l6)* >>> * **Linked** count -8 (l0 -> l7)* >>> * **Linked** count -9 (l0 -> l8)* >>> * **Linked** count -10 (l0 -> l9)* >>> * >>> * >>> real 0m0.056s >>> user 0m0.032s >>> sys 0m0.023s >>> >>> Hence in my understanding Python is better in terms of execution time >>> when compared with PyPy. >>> >>> Kindly suggest me if my trails are proper and why PyPy failed to achieve >>> better speed then PyPy ? >>> >>> Is this a know issue or whether we have any fix for this ? >>> >>> Thanking you, >>> -- >>> Sasikanth >>> >>> >>> >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >> PyPy's JIT works by optimizing loops and functions that are frequently >> run, this generally means your code needs to run at least .5 seconds for >> the JIT to kick in and show benefits. Your code contains no loops for the >> JIT to optimize. >> >> Alex >> >> -- >> "I disapprove of what you say, but I will defend to the death your right >> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) >> "The people's good is the highest law." -- Cicero >> >> > > > > > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From lizfreudber at immk.net Fri May 18 18:45:40 2012 From: lizfreudber at immk.net (Liz Freudberg) Date: Fri, 18 May 2012 17:45:40 +0100 Subject: [pypy-dev] (Web Position / Analysis) Message-ID: <58098328.20120518174540@immk.net> pypy-dev at codespeak.net : Our company can place your web site in the top positions of the major search engines. Want a free estimate? Reply to us if you are interested in getting top positioning in the major search engines and we will give you a free analysis of your site. Please include the best way to reach you. Liz Freudberg -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sat May 19 20:05:34 2012 From: arigo at tunes.org (Armin Rigo) Date: Sat, 19 May 2012 20:05:34 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi all, We started working on it a few days ago and it's already progressed well: http://github.com/arigo/ffi A bient?t, Armin. From fijall at gmail.com Mon May 21 15:45:07 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 21 May 2012 15:45:07 +0200 Subject: [pypy-dev] New buildslave Message-ID: Hi Thanks to the Allegro group and ?ukasz Langa, we got a new powerful buildslave. We cannot run benchmarks there, since we have to share the machine with cPython buildbots, however, we can run all other stuff which should take some load off tannit. I did not set it up yet in buildmaster since I'm pondering about locks and how to do it now. Feel free to comment. The server admin is Lukasz, to complain or praise write to - lukasz.langa at allegro.pl Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From rorsoft at gmail.com Wed May 23 06:07:19 2012 From: rorsoft at gmail.com (bookaa) Date: Wed, 23 May 2012 12:07:19 +0800 Subject: [pypy-dev] bug with MinGW32 Message-ID: I download last version of PyPy source from https://github.com/pypy/pypy in MinGW32 (mingw-get-inst-20120426.exe), run: pypy/bin/py.py --cc=mingw32-gcc get error: pypy.translator.platform.CompilationError: CompilationError(err=""" e:\tem\usession-unknown-1\gcctest.c: In function 'main': e:\tem\usession-unknown-1\gcctest.c:25:51: error: 'uintptr_t' undeclared (first use in this function) this is how I fix this bug: diff -crN pypy\rpython\tool\rfficache.py pypy\rpython\tool\rfficache.bookaa.py *** pypy\rpython\tool\rfficache.py Mon May 21 14:45:32 2012 --- pypy\rpython\tool\rfficache.bookaa.py Wed May 23 08:07:33 2012 *************** *** 13,19 **** from pypy.tool.gcc_cache import build_executable_cache def ask_gcc(question, add_source=""): ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h'] if os.name != 'nt': includes += ['inttypes.h'] include_string = "\n".join(["#include <%s>" % i for i in includes]) --- 13,19 ---- from pypy.tool.gcc_cache import build_executable_cache def ask_gcc(question, add_source=""): ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h', 'stdint.h'] if os.name != 'nt': includes += ['inttypes.h'] include_string = "\n".join(["#include <%s>" % i for i in includes]) diff -crN pypy\rlib\rposix.py pypy\rlib\rposix.bookaa.py *** pypy\rlib\rposix.py Mon May 21 14:45:32 2012 --- pypy\rlib\rposix.bookaa.py Wed May 23 08:29:36 2012 *************** *** 24,29 **** --- 24,30 ---- separate_module_sources =[''' /* Lifted completely from CPython 3.3 Modules/posix_module.c */ #include /* for _msize */ + #include /* for intptr_t */ typedef struct { intptr_t osfhnd; char osfile; -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Wed May 23 06:29:40 2012 From: lac at openend.se (Laura Creighton) Date: Wed, 23 May 2012 06:29:40 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: Message from Armin Rigo of "Tue, 15 May 2012 16:23:14 +0200." References: Message-ID: <201205230429.q4N4TebX003789@theraft.openend.se> In a message of Tue, 15 May 2012 16:23:14 +0200, Armin Rigo writes: tons of things that I agree with more or less wholesale, though I think that there is a slightly larger market for Cython in the immediate future ... Then comes this: >The simplest FFI we know of for a high-level language is LuaJIT's FFI. > If you are interested, read the four pages starting at >http://luajit.org/ext_ffi.html . A crucial design decision was to >avoid making use of tons of objects representing the C types; instead, >the user mostly specifies C types as plain strings, simplifying the >basic API a lot. > >So we would like to propose some design very similar to that one. The >basic public API looks more or less fine to me. There are of course >preferences and topics to discuss, but let's leave that for later. >Most importantly, we would welcome a system where the C declarations >can be parsed either internally or (for more complicated cases) >externally, with the help of a real C compiler. This might include >cases like macros, where the user would need to write (as C code) a >small function calling the macro, and have it compiled when he imports >his Python module. more cool stuff. Sounds great to me. Except that Mike Pell (LuaJit's author) hasn't been cc'd on this note. And so what I would like to know before backing this new idea as clearly the right thing to do, is some statement from him that he is happy with it. Because, dear God, if he had not already existed and made this thing, then *we* would be the people with the best working FFI. And before somebody decided to do things 'our way' (Whichever of the 2 I am thinking of) we would dearly love to get a chance to shout at them -- No NO NO -- our way was an experiment and too messy to be used. The string approach is not messy in the same way as the let-us-make-enough-objects approach is. But unpacking them might lead to other messiness. I'd dearly love to hear from those who use this interface and LuaJIT all the time to hear what they think of it, it's greatest strengths and limitations. But JuaJIT has a nice public mailing list -- why not let them know we are planning to do this and see what they think? Mike Pall I think would be interested and would warn us about problems he has found. Laura From lac at openend.se Wed May 23 06:47:23 2012 From: lac at openend.se (Laura Creighton) Date: Wed, 23 May 2012 06:47:23 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: Message from Laura Creighton of "Wed, 23 May 2012 06:29:40 +0200." <201205230429.q4N4TebX003789@theraft.openend.se> References: <201205230429.q4N4TebX003789@theraft.openend.se> Message-ID: <201205230447.q4N4lNQJ006239@theraft.openend.se> >Sounds great to me. Except that Mike Pell (LuaJit's author) hasn't >been cc'd on this note. And so what I would like to know before Ooops, it's Mike Pall. No disrespect intended (and indeed I spelt the name correctly the second time it was used in my reply). That was a typo. I apologise. Laura From fijall at gmail.com Wed May 23 10:00:59 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 23 May 2012 10:00:59 +0200 Subject: [pypy-dev] bug with MinGW32 In-Reply-To: References: Message-ID: Hi I don't know about the github mirror, the official one is on bitbucket. Cheers,\ fijal On Wed, May 23, 2012 at 6:07 AM, bookaa wrote: > ** > > I download last version of PyPy source from https://github.com/pypy/pypy > in MinGW32 (mingw-get-inst-20120426.exe), run: > pypy/bin/py.py --cc=mingw32-gcc > get error: > pypy.translator.platform.CompilationError: CompilationError(err=""" > e:\tem\usession-unknown-1\gcctest.c: In function 'main': > e:\tem\usession-unknown-1\gcctest.c:25:51: error: 'uintptr_t' > undeclared (first use in this function) > > this is how I fix this bug: > > diff -crN pypy\rpython\tool\rfficache.py pypy\rpython\tool\ > rfficache.bookaa.py > *** pypy\rpython\tool\rfficache.py Mon May 21 14:45:32 2012 > --- pypy\rpython\tool\rfficache.bookaa.py Wed May 23 08:07:33 2012 > *************** > *** 13,19 **** > from pypy.tool.gcc_cache import build_executable_cache > > def ask_gcc(question, add_source=""): > ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h'] > if os.name != 'nt': > includes += ['inttypes.h'] > include_string = "\n".join(["#include <%s>" % i for i in includes]) > --- 13,19 ---- > from pypy.tool.gcc_cache import build_executable_cache > > def ask_gcc(question, add_source=""): > ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h', 'stdint.h'] > if os.name != 'nt': > includes += ['inttypes.h'] > include_string = "\n".join(["#include <%s>" % i for i in includes]) > diff -crN pypy\rlib\rposix.py pypy\rlib\rposix.bookaa.py > *** pypy\rlib\rposix.py Mon May 21 14:45:32 2012 > --- pypy\rlib\rposix.bookaa.py Wed May 23 08:29:36 2012 > *************** > *** 24,29 **** > --- 24,30 ---- > separate_module_sources =[''' > /* Lifted completely from CPython 3.3 Modules/posix_module.c */ > #include /* for _msize */ > + #include /* for intptr_t */ > typedef struct { > intptr_t osfhnd; > char osfile; > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sasikanth.ece at gmail.com Wed May 23 10:20:30 2012 From: sasikanth.ece at gmail.com (Sasikanth Eda) Date: Wed, 23 May 2012 13:50:30 +0530 Subject: [pypy-dev] Need your suggestions in thoughts of improving performance of REST based cgi-server Message-ID: Hai All, I have tried to perform performance testing of a typical CGI-server Rest based SNIA CDMI specifications and the results are as follows; Python PyPy 16 KB (Object) PUT 196 ms 325 ms 64 KB (Object) PUT 176 ms 396 ms 1 MB (Object) PUT 248 ms 927 ms Container PUT 121 ms 317 ms where in each scenario python is faster than PyPy. I request your inputs/suggestion techniques towards optimizing such load loads. Kindly share your experiences and feedback in this scenario. -- Thanking you, Sasikanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed May 23 10:23:09 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 23 May 2012 10:23:09 +0200 Subject: [pypy-dev] Need your suggestions in thoughts of improving performance of REST based cgi-server In-Reply-To: References: Message-ID: Hi If you reload the python interpreter for each CGI request, the JIT has absolutely no chance to warm up. You need some sort of FastCGI solution where the interpreter stays in the memory between requests On Wed, May 23, 2012 at 10:20 AM, Sasikanth Eda wrote: > > Hai All, > > I have tried to perform performance testing of a typical CGI-server Rest > based SNIA CDMI specifications and the results are as follows; > > Python PyPy > 16 KB (Object) PUT 196 ms 325 ms > 64 KB (Object) PUT 176 ms 396 ms > 1 MB (Object) PUT 248 ms 927 ms > Container PUT 121 ms 317 ms > > where in each scenario python is faster than PyPy. > > I request your inputs/suggestion techniques towards optimizing such load > loads. > > Kindly share your experiences and feedback in this scenario. > > -- > Thanking you, > Sasikanth > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From piotr.skamruk at gmail.com Wed May 23 10:40:34 2012 From: piotr.skamruk at gmail.com (Piotr Skamruk) Date: Wed, 23 May 2012 10:40:34 +0200 Subject: [pypy-dev] Need your suggestions in thoughts of improving performance of REST based cgi-server In-Reply-To: References: Message-ID: Probably You can easily convert your application into wsgi compilant, which can be run in long time running process (for example under http://pypi.python.org/pypi/waitress/ server). In that situation Your code runned under pypy could "warm up", and so - can be optimized in runtime. Could You post somewhere source of your cgi-server? I can help You with such conversion... I'm also curious about how much that will help in both situations, i mean under cpython, and under pypy. btw. which version of cpython You have in mind under "Python" name? And which version of pypy You are using? From arigo at tunes.org Wed May 23 12:49:06 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 23 May 2012 12:49:06 +0200 Subject: [pypy-dev] bug with MinGW32 In-Reply-To: References: Message-ID: Hi Bookaa, On Wed, May 23, 2012 at 6:07 AM, bookaa wrote: > !???? includes = ['stdlib.h', 'stdio.h', 'sys/types.h', 'stdint.h'] The MSVC compiler before 2010 doesn't provide a 'stdint.h', so this change would break it. You need to come up with a more complicated solution. Armin From tbaldridge at gmail.com Wed May 23 14:44:07 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Wed, 23 May 2012 07:44:07 -0500 Subject: [pypy-dev] Jit hints from python Message-ID: While looking at ways to optimize the Clojure-Py code (http://github.com/halgari/clojure-py) for pypy, I'm noticing that much of Clojure-Py could benefit greatly from giving purefunction hints to the JIT. For example, let's look at the polymorphic Clojure functions known as "protocols": def ProtocolFn(object): def __init__(self): self.d = {} def extend(self, tp, fn): self.d[tp] = fn def getFn(self, tp): return self.d[tp] def __call__(self, *args, **kw): return self.getFn(type(args[0]))(*args **kw) Now, if I was writing this in RPython I would have extend clone the dictionary each time, then makr getFn as "pure" and suddenly I'd expect to see the JIT start in-lining calls to this ProtocolFn, and I would also expect to see a rather major performance boost. Also, since the majority of Clojure code deals with immutable data structures, I could actually mark 80% of my data structure functions as pure. So my question is, how easy/hard would it be to somehow get a @purefunction decorator in Python instead of just RPython? For that matter, having __immutable_fields__ would also be nice, but that may be asking for too much. Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From rorsoft at gmail.com Wed May 23 14:46:44 2012 From: rorsoft at gmail.com (bookaa) Date: Wed, 23 May 2012 20:46:44 +0800 Subject: [pypy-dev] bug with MinGW32 In-Reply-To: References: Message-ID: Re Armin and fijal I download last version of PyPy source pypy-pypy-4a38b43757e3.zip from https://bitbucket.org/pypy/pypy in MinGW32 (mingw-get-inst-20120426.exe), run: pypy/bin/py.py --cc=mingw32-gcc get error: pypy.translator.platform.CompilationError: CompilationError(err=""" e:\tem\usession-default-0\gcctest.c: In function 'main': e:\tem\usession-default-0\gcctest.c:25:51: error: 'uintptr_t' undeclared (first use in this function) this is how I fix this bug: diff -crN pypy-pypy-4a38b43757e3/pypy/rlib/rposix.py pypy-pypy-4a38b43757e3.bookaa/pypy/rlib/rposix.py *** pypy-pypy-4a38b43757e3/pypy/rlib/rposix.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/rlib/rposix.py Wed May 23 20:25:10 2012 *************** *** 75,80 **** --- 75,85 ---- return 0; } ''',] + from pypy.translator import platform + if platform.platform.cc.startswith('mingw32'): + separate_module_sources[0] = """ + #include /* for intptr_t */ + """ + separate_module_sources[0] export_symbols = ['_PyVerify_fd'] else: separate_module_sources = [] diff -crN pypy-pypy-4a38b43757e3/pypy/rpython/tool/rfficache.py pypy-pypy-4a38b43757e3.bookaa/pypy/rpython/tool/rfficache.py *** pypy-pypy-4a38b43757e3/pypy/rpython/tool/rfficache.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/rpython/tool/rfficache.py Wed May 23 20:38:08 2012 *************** *** 13,19 **** from pypy.tool.gcc_cache import build_executable_cache def ask_gcc(question, add_source=""): ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h'] if os.name != 'nt': includes += ['inttypes.h'] include_string = "\n".join(["#include <%s>" % i for i in includes]) --- 13,23 ---- from pypy.tool.gcc_cache import build_executable_cache def ask_gcc(question, add_source=""): ! from pypy.translator import platform ! if platform.platform.cc.startswith('mingw32'): ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h', 'stdint.h'] ! else: ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h'] if os.name != 'nt': includes += ['inttypes.h'] include_string = "\n".join(["#include <%s>" % i for i in includes]) diff -crN pypy-pypy-4a38b43757e3/pypy/translator/platform/posix.py pypy-pypy-4a38b43757e3.bookaa/pypy/translator/platform/posix.py *** pypy-pypy-4a38b43757e3/pypy/translator/platform/posix.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/translator/platform/posix.py Wed May 23 20:41:12 2012 *************** *** 55,60 **** --- 55,62 ---- if relto: response_file = relto.bestrelpath(response_file) + if self.cc.startswith('mingw32'): + return ["-Wl,--export-all-symbols,--version-script=%s" % (response_file,)] return ["-Wl,--export-dynamic,--version-script=%s" % (response_file,)] def _link(self, cc, ofiles, link_args, standalone, exe_name): -------------- next part -------------- A non-text attachment was scrubbed... Name: bookaa.dif Type: application/octet-stream Size: 2507 bytes Desc: not available URL: From amauryfa at gmail.com Wed May 23 15:26:58 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 23 May 2012 15:26:58 +0200 Subject: [pypy-dev] bug with MinGW32 In-Reply-To: References: Message-ID: 2012/5/23 bookaa > if platform.platform.cc.**startswith('mingw32'): > This is better written as: from pypy.translator.platform import platform if platform.name == 'minwg32' See rlib/libffi.py for an example. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbaldridge at gmail.com Thu May 24 13:13:06 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Thu, 24 May 2012 06:13:06 -0500 Subject: [pypy-dev] Faster PyPy translation (for debugging) Message-ID: I'm working on writing a module for PyPy, the code runs fine via py.py, but I'm still debugging the RTyping errors. What can I do to speed up the translation process. Currently I'm running pypy translate.py --opt=0 targetpypystandalone.py On my machine that takes about 800sec before it starts rtyping my module. Are there some better options I can use if I simply want to make sure that my module will fit into the PyPy typesystem? Thanks, Timothy From amauryfa at gmail.com Thu May 24 13:41:55 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 24 May 2012 13:41:55 +0200 Subject: [pypy-dev] Faster PyPy translation (for debugging) In-Reply-To: References: Message-ID: Hi, 2012/5/24 Timothy Baldridge > pypy translate.py --opt=0 targetpypystandalone.py > > On my machine that takes about 800sec before it starts rtyping my > module. Are there some better options I can use if I simply want to > make sure that my module will fit into the PyPy typesystem? > I often add the options --no-allworkingmodules --withmod-MYMODULE (after the target name) to remove modules not strictly necessary to the Python interpreter. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Thu May 24 16:24:52 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 24 May 2012 07:24:52 -0700 (PDT) Subject: [pypy-dev] Faster PyPy translation (for debugging) In-Reply-To: References: Message-ID: Hi Timothy, > On my machine that takes about 800sec before it starts rtyping my > module. Are there some better options I can use if I simply want to > make sure that my module will fit into the PyPy typesystem? if it can find the errors, then the fastest is to use: pypy/bin/checkmodule.py but I very rarely find errors that way. Second is to write a jit test, which does partial translation (only of the code reachable from the test) and can use a fake space, which really speeds things up. Look at any of the jit tests themselves or those in module (*/test/*jit*.py) for some examples. It's a bit of work (and I didn't write mine, but got it from one of the core pypy folks :) ), but I find it in practice by far the quickest way to find rtyping errors (albeit, again, not always conclusive). HTH, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From fwierzbicki at gmail.com Thu May 24 18:17:10 2012 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 24 May 2012 09:17:10 -0700 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] Message-ID: On Thu, May 24, 2012 at 7:24 AM, wrote: > if it can find the errors, then the fastest is to use: > > ? pypy/bin/checkmodule.py > > but I very rarely find errors that way. Second is to write a jit test, > which does partial translation (only of the code reachable from the test) > and can use a fake space, which really speeds things up. Look at any of the > jit tests themselves or those in module (*/test/*jit*.py) for some examples. > It's a bit of work (and I didn't write mine, but got it from one of the > core pypy folks :) ), but I find it in practice by far the quickest way to > find rtyping errors (albeit, again, not always conclusive). I've been meaning to dip a toe in PyPy but had trouble getting the JVM translator to work the last time I tried. How hard would it be to use a fake space (or some other technique) to just spit out a parser for the JVM? -Frank From tbaldridge at gmail.com Thu May 24 18:19:27 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Thu, 24 May 2012 11:19:27 -0500 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] In-Reply-To: References: Message-ID: > I've been meaning to dip a toe in PyPy but had trouble getting the JVM > translator to work the last time I tried. How hard would it be to use > a fake space (or some other technique) to just spit out a parser for > the JVM? Side note, I would really love to see the JVM backend more maintained. If I could write Clojure in RPython, then run it on the JVM...with a full tracing JIT, that would just be insanely awesome. Timothy From fwierzbicki at gmail.com Thu May 24 18:23:41 2012 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 24 May 2012 09:23:41 -0700 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] In-Reply-To: References: Message-ID: On Thu, May 24, 2012 at 9:19 AM, Timothy Baldridge wrote: > Side note, I would really love to see the JVM backend more maintained. > If I could write Clojure in RPython, then run it on the JVM...with a > full tracing JIT, that would just be insanely awesome. Holy turtles all the way down batman! :) -Frank From michal at bendowski.pl Thu May 24 18:31:20 2012 From: michal at bendowski.pl (Michal Bendowski) Date: Thu, 24 May 2012 18:31:20 +0200 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] In-Reply-To: References: Message-ID: Just two add my two cents: I made the JVM backend translate the standard interpreter in the jvm-improvements branch a few months ago (just a few changes). The branch has been closed though and I had a busy time so I didn't do anything about it. For the last few weeks/months I was sitting in my cave and working on my thesis, which now lets you use JVM classes directly in RPython (rjvm) and also contains a naive, reflection-based jvm module for the resulting interpreter. I know this is not in the OSS spirit, but it so happened, sorry. I'm planning to merge my changes soon and hopefully keep on working on the JVM backend. Micha? On Thu, May 24, 2012 at 6:23 PM, fwierzbicki at gmail.com wrote: > On Thu, May 24, 2012 at 9:19 AM, Timothy Baldridge wrote: > >> Side note, I would really love to see the JVM backend more maintained. >> If I could write Clojure in RPython, then run it on the JVM...with a >> full tracing JIT, that would just be insanely awesome. > Holy turtles all the way down batman! :) > > -Frank > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From fwierzbicki at gmail.com Thu May 24 18:42:03 2012 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 24 May 2012 09:42:03 -0700 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] In-Reply-To: References: Message-ID: On Thu, May 24, 2012 at 9:31 AM, Michal Bendowski wrote: > Just two add my two cents: I made the JVM backend translate the > standard interpreter in the jvm-improvements branch a few months ago > (just a few changes). The branch has been closed though and I had a > busy time so I didn't do anything about it. > > For the last few weeks/months I was sitting in my cave and working on > my thesis, which now lets you use JVM classes directly in RPython > (rjvm) and also contains a naive, reflection-based jvm module for the > resulting interpreter. I know this is not in the OSS spirit, but it so > happened, sorry. I'm planning to merge my changes soon and hopefully > keep on working on the JVM backend. Nice! Can't wait to see it - and of course I understand that working on a thesis makes people disappear :) -Frank From fijall at gmail.com Thu May 24 19:00:48 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 24 May 2012 19:00:48 +0200 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] In-Reply-To: References: Message-ID: On Thu, May 24, 2012 at 6:19 PM, Timothy Baldridge wrote: > > I've been meaning to dip a toe in PyPy but had trouble getting the JVM > > translator to work the last time I tried. How hard would it be to use > > a fake space (or some other technique) to just spit out a parser for > > the JVM? > > > Side note, I would really love to see the JVM backend more maintained. > If I could write Clojure in RPython, then run it on the JVM...with a > full tracing JIT, that would just be insanely awesome. > > Timothy > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Note that you don't get the full tracing JIT for free with JVM. As in the RPython implementation at the first approximation would just directly translate to JVM bytecode and be happy (and slow). To add a tracing JIT for the JVM you need some significant work, see antonio cuni's thesis for his work on .NET Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Thu May 24 19:08:53 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 24 May 2012 19:08:53 +0200 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] In-Reply-To: References: Message-ID: <4FBE6B25.6030205@gmail.com> Hello Michal, On 05/24/2012 06:31 PM, Michal Bendowski wrote: > Just two add my two cents: I made the JVM backend translate the > standard interpreter in the jvm-improvements branch a few months ago > (just a few changes). The branch has been closed though and I had a > busy time so I didn't do anything about it. the branch has been merged, IIRC. Last time I tried, basic translation to jvm worked. However, we never set up a nighlt translation, so things might have broken in the meantime :-( > For the last few weeks/months I was sitting in my cave and working on > my thesis, which now lets you use JVM classes directly in RPython > (rjvm) and also contains a naive, reflection-based jvm module for the > resulting interpreter. I know this is not in the OSS spirit, but it so > happened, sorry. I'm planning to merge my changes soon and hopefully > keep on working on the JVM backend. wow, that's a very nice news. I'd really like to see it. A working rjvm module is also a big step forward in having a JIT for the JVM, although the road is still long. From arigo at tunes.org Thu May 24 19:43:16 2012 From: arigo at tunes.org (Armin Rigo) Date: Thu, 24 May 2012 19:43:16 +0200 Subject: [pypy-dev] Jit hints from python In-Reply-To: References: Message-ID: Hi Timothy, On Wed, May 23, 2012 at 2:44 PM, Timothy Baldridge wrote: > So my question is, how easy/hard would it be to somehow get a > @purefunction decorator in Python instead of just RPython? For that > matter, having __immutable_fields__ would also be nice, but that may > be asking for too much. It's the opposite. We could more or less easily have an "__immutable_fields__" declaration that lets you write to an attribute of a class only once. Maybe call it "__immutable_slots__" and have it work like "__slots__". But providing a @purefunction or @elidable decorator at the level of Python raises more questions. For example, I'm sure that the intended meaning is that if the function is called as f(10) with a constant 10, then we can constant-fold it. But does this mean the value 10, or the object 10 (which might be different even though the value 10 is the same)? Or do you want to go all the way and say that as soon as Python objects compare equal, then the function call can be constant-folded? Do you want to require the arguments to be hashable objects, so that we can use a dictionary to know the result? And anyway, what does "compare equal as Python objects" mean at the level of the JIT? Nothing... So your original suggestion is hard to define correctly... Note that already nowadays you can hack (maybe we should just make such hacks more explicit): if you use not a general-purpose dictionary but instead a namespace like a module or a class, then reads are constant-folded with an out-of-line guard. This means you could rewrite your code for example like this: class ProtocolFn(object): def __init__(self): self.m = new.module('protocol') def extend(self, tp, fn): setattr(self.m, tp, fn) def getFn(self, tp): return getattr(self.m, tp) Or even "self.d = new.module('protocol').__dict__", with no changes in the rest of the code. I'm not 100% sure if it works, but if you're successful at using it, then we could provide it more officially, e.g. with "import __pypy__; d = __pypy__.rarely_changing_dict()" :-) A bient?t, Armin. From tbaldridge at gmail.com Thu May 24 20:07:38 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Thu, 24 May 2012 13:07:38 -0500 Subject: [pypy-dev] Jit hints from python In-Reply-To: References: Message-ID: > Note that already nowadays you can hack (maybe we should just make > such hacks more explicit): if you use not a general-purpose dictionary > but instead a namespace like a module or a class, then reads are > constant-folded with an out-of-line guard. ?This means you could > rewrite your code for example like this: You're right, that gives me a bout a 5x speedup from by micro-benchmarks. The only problem is that module and class only accept strings as attribute keys. So we have to do something like type(args[0]).__name__ which is kindof hack-ish. At any rate, I'm playing around with writing Clojure as a pypy module. Last night (and this morning) I got this class (PolymorphicFn) implemented as an interpreter level object with a App level Typedef. Clojure defines several function types that would see a rather good performance gain from being more tightly integrated with the interpreter. For example, Clojure defines multi-arity functions, variadic functions, multi-methods, etc. The performance gains of handling these on an interpreter level interest me, so I'm going to see where this rabbit hole leads. Timothy From ralphdinkin at caryon.com Fri May 25 00:39:57 2012 From: ralphdinkin at caryon.com (Ralph Dinkins) Date: Thu, 24 May 2012 23:39:57 +0100 Subject: [pypy-dev] +Site Place/ Analysis/ Offer Message-ID: <51648799.20120524233957@caryon.com> pypy-dev at codespeak.net : Your site has a good look and feel but you're not getting as much web traffic as you could be getting. Can we show you how to change that? It won't cost you anything for us to review your online business and the results could spike your web revenue. Reply to us today and we will do a free assessment of your web presence. Don't forget to include the best way we can contact you. Sincerely, Ralph Dinkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjenvey at underboss.org Fri May 25 01:07:53 2012 From: pjenvey at underboss.org (Philip Jenvey) Date: Thu, 24 May 2012 16:07:53 -0700 Subject: [pypy-dev] Translating just the parser for JVM [was: Faster PyPy translation (for debugging)] In-Reply-To: <4FBE6B25.6030205@gmail.com> References: <4FBE6B25.6030205@gmail.com> Message-ID: On May 24, 2012, at 10:08 AM, Antonio Cuni wrote: > Hello Michal, > > On 05/24/2012 06:31 PM, Michal Bendowski wrote: >> Just two add my two cents: I made the JVM backend translate the >> standard interpreter in the jvm-improvements branch a few months ago >> (just a few changes). The branch has been closed though and I had a >> busy time so I didn't do anything about it. > > the branch has been merged, IIRC. Last time I tried, basic translation to jvm > worked. However, we never set up a nighlt translation, so things might have > broken in the meantime :-( You probably still have to force translation in 32 bit mode. e.g. on OS X: arch -i386 python translate.py etc -- Philip Jenvey From rorsoft at gmail.com Fri May 25 02:35:08 2012 From: rorsoft at gmail.com (bookaa) Date: Fri, 25 May 2012 08:35:08 +0800 Subject: [pypy-dev] bug with MinGW32 In-Reply-To: References: Message-ID: yes, it should be like this. thank you Bookaa From: Amaury Forgeot d'Arc Sent: Wednesday, May 23, 2012 9:26 PM To: bookaa Cc: pypy-dev at python.org Subject: Re: [pypy-dev] bug with MinGW32 2012/5/23 bookaa if platform.platform.cc.startswith('mingw32'): This is better written as: from pypy.translator.platform import platform if platform.name == 'minwg32' See rlib/libffi.py for an example. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri May 25 11:03:42 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 25 May 2012 11:03:42 +0200 Subject: [pypy-dev] Jit hints from python In-Reply-To: References: Message-ID: Hi Timothy, On Thu, May 24, 2012 at 8:07 PM, Timothy Baldridge wrote: > The only problem is that module and class only > accept strings as attribute keys. Just a quick note: that's true for classes but not for modules. Armin From cfbolz at gmx.de Fri May 25 11:14:59 2012 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 25 May 2012 11:14:59 +0200 Subject: [pypy-dev] Jit hints from python In-Reply-To: References: Message-ID: <4FBF4D93.2020504@gmx.de> On 05/25/2012 11:03 AM, Armin Rigo wrote: > Hi Timothy, > > On Thu, May 24, 2012 at 8:07 PM, Timothy Baldridge wrote: >> The only problem is that module and class only >> accept strings as attribute keys. > > Just a quick note: that's true for classes but not for modules. Yes, but that doesn't help anyway: When you use non-string keys, the module dict implementation reverts to a normal one. Cheers, Carl Friedrich From matti.picus at gmail.com Fri May 25 11:54:12 2012 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 25 May 2012 12:54:12 +0300 Subject: [pypy-dev] bug with MinGW32 In-Reply-To: References: Message-ID: <4FBF56C4.9030407@gmail.com> On 25/05/2012 3:35 AM, bookaa wrote: > yes, it should be like this. > thank you > Bookaa > > *From:* Amaury Forgeot d'Arc > *Sent:* Wednesday, May 23, 2012 9:26 PM > *To:* bookaa > *Cc:* pypy-dev at python.org > *Subject:* Re: [pypy-dev] bug with MinGW32 > > 2012/5/23 bookaa > > > if platform.platform.cc > .startswith('mingw32'): > > > This is better written as: > from pypy.translator.platform import platform > if platform.name == 'minwg32' > See rlib/libffi.py for an example. > > -- > Amaury Forgeot d'Arc > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev Thanks bookaa. I committed this to the default branch, also allowing cc=gcc, i.e. --cc=gcc or $export CC=gcc Support for mingw seems not to be complete: $/c/Python27/python pytest.py pypy/rlib/test/test_rposix.py shows failing tests with cc set via environment variable CC. Matti From mmueller at python-academy.de Fri May 25 12:12:52 2012 From: mmueller at python-academy.de (=?ISO-8859-15?Q?Mike_M=FCller?=) Date: Fri, 25 May 2012 12:12:52 +0200 Subject: [pypy-dev] PyPy Sprint in Leipzig is approaching Message-ID: <4FBF5B24.3060108@python-academy.de> Hi all, It is only four weeks till the sprint in Leipzig. So it is not too earlier to get started with travel planning. Details about the sprint are here: http://morepypy.blogspot.de/2012/04/pypy-sprint-in-leipzig-june-22-27.html If you plan to attend please add you name here: https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leipzig2012/people.txt Here is a short list of hotels nearby: http://www.python-academy.com/center/accommodation.html You can also stay in the city center. Public transportation is fast and frequent. It should take about 25 - 30 minutes from a inner city hotel to the sprint location. You can ask me if you have any questions about location or accommodation. Cheers, Mike From arigo at tunes.org Fri May 25 20:44:01 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 25 May 2012 20:44:01 +0200 Subject: [pypy-dev] buildmaster moved Message-ID: Hi all, Wyvern, which was the machine serving a few PyPy-related services --- most notably the buildmaster and the nightly builds at buildbot.pypy.org --- more or less died a few days ago. It was after a long life: we had the server around 2005, and now the particular small piece that broke down is no longer found. For information the hard drive was broken once already and is not as old as the rest. So David and remotely Antonio moved the HD inside another machine and reconfigured the DNS. All services should be back on-line now. Thanks to them :-) If you're running a buildslave, please make sure that its file buildbot.tac points to "buildbot.pypy.org" and not explicitly to "wyvern.cs.uni-duesseldorf.de". The latter no longer points to anything. A bient?t, Armin. From tbaldridge at gmail.com Fri May 25 22:48:26 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Fri, 25 May 2012 15:48:26 -0500 Subject: [pypy-dev] Differences between app and interp level calling methods Message-ID: For my clojure module, I'm trying to figure out if I should go with app level or interp level code. Many of my classes will do dispatching on __call__ or the equivalent. Are interpreter level variadic functions going to be faster (via call_args) vs an app level __call__(self, args_w) ? A bit more background here, Clojure functions dispatch on the number of arguments: (defn + [x] x [x y] (add x y)) [x y & rest] (reduce + (add x y) rest)) For something like this, am I going to see much of a performance boost by dropping to the interpreter level from the dispatch between these three sub-function types? Thanks, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From rorsoft at gmail.com Sat May 26 03:55:18 2012 From: rorsoft at gmail.com (bookaa) Date: Sat, 26 May 2012 09:55:18 +0800 Subject: [pypy-dev] add --cc option to pytest Message-ID: <68622C6A999E45D0A12075506A6EB2A2@vSHliutaotao> according to http://web.archiveorange.com/archive/v/cZ0K2Jye2cyGARzh9OlA after patch pypy/conftest.py, pytest.py now support --cc option: pytest.py --cc=mingw32-gcc pypy/translator/test/test_exceptiontransform.py but we still can not use --cc option on specify function: pytest.py --cc=mingw32-gcc pypy/translator/test/test_exceptiontransform.py::TestOOType::()::test_simple this is how to add this support: *** pypy-pypy-4a38b43757e3\_pytest\config.py Sat May 26 09:16:32 2012 --- pypy-pypy-4a38b43757e3.bookaa\_pytest\config.py Sat May 26 09:26:36 2012 *************** *** 157,162 **** --- 157,164 ---- for arg in args + [current]: if hasattr(arg, 'startswith') and arg.startswith("--"): continue + if arg.find('::') != -1: + arg = arg[:arg.find('::')] anchor = current.join(arg, abs=1) if anchor.check(): # we found some file object self._path2confmods[None] = self.getconftestmodules(anchor) now we can do this: pytest.py --cc=mingw32-gcc pypy/translator/test/test_exceptiontransform.py::TestOOType::()::test_simple and this pypy>test_all.py --cc=mingw32-gcc translator/test/test_exceptiontransform.py::TestOOType::()::test_simple but, I can not pass the test, with or withou --cc option. Error message: E AssertionError: ilasm failed to assemble (ilasm): E E Microsoft (R) .NET Framework IL Assembler. Version 2.0.50727.4927 E Copyright (c) Microsoft Corporation. All rights reserved. E Assembling 'e:\tem\usession-default-137\main.il' to EXE --> 'e:\tem\usession-default-137\main.exe' E Source file is ANSI E E ***** FAILURE ***** E E e:\tem\usession-default-137\main.il(161) : error : syntax error at token '[' in: call [mscorlib]System.Object rpyexc_fetch_value() any advice ? thanks, Bookaa -------------- next part -------------- An HTML attachment was scrubbed... URL: From rorsoft at gmail.com Sat May 26 15:47:31 2012 From: rorsoft at gmail.com (bookaa) Date: Sat, 26 May 2012 21:47:31 +0800 Subject: [pypy-dev] pytest test_rposix.py with mingw32 Message-ID: <5E5F88FFB39147BB8A38143E94DA595F@vSHliutaotao> keywords: _PyVerify_fd __pioinfo _msize PyPy can not pass this test: pytest.py --cc=mingw32-gcc pypy\rlib\test\test_rposix.py::TestPosixUnicode::()::test_is_valid_fd After 2 days of hard work, I finally realise why. Patch file attached. The problem is in _PyVerify_fd: FILE* f = fopen("d:\\55.txt","w"); int no = fileno(f); int flg1 = _PyVerify_fd(no); fclose(f); int flg2 = _PyVerify_fd(no); printf("flg1 %d should be 1, flg2 %d should be 0", flg1, flg2) in file rposix.py, source of function _PyVerify_fd is write to a c file, and will be compile to a Windows DLL file. In this function, it use extern __declspec(dllimport) char * __pioinfo[]; __pioinfo as a global variable. I find use __pioinfo in a DLL file is not safe. the __pioinfo and _msize may export from many DLL: msvcrt.dll msvcr90.dll msvcr90d.dll or maybe msvcr70.dll ... If we fopen and fileno in a EXE file, and call _PyVerify_fd in a DLL file, its very danger. The EXE and DLL may reference different __pioinfo and get error. So I think test_rposix.py::TestPosixUnicode::()::test_is_valid_fd can pass in MSVC is only a coincidence. _PyVerify_fd in a DLL and use dllimport __pioinfo is not safe. In my fix, I do not assume any DLL we should use. I try to find the current DLL which export __pioinfo already loaded. Sure this is not very good. But I think if we must place _PyVerify_fd in a DLL, no good solution. After this path, this tests are pass: pytest.py --cc=mingw32-gcc pypy\rlib\test\test_rposix.py pytest.py pypy\rlib\test\test_rposix.py ----------- diff -crN pypy-pypy-4a38b43757e3/_pytest/config.py pypy-pypy-4a38b43757e3.bookaa/_pytest/config.py *** pypy-pypy-4a38b43757e3/_pytest/config.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/_pytest/config.py Sat May 26 20:47:43 2012 *************** *** 157,162 **** --- 157,164 ---- for arg in args + [current]: if hasattr(arg, 'startswith') and arg.startswith("--"): continue + if arg.find('::') != -1: + arg = arg[:arg.find('::')] anchor = current.join(arg, abs=1) if anchor.check(): # we found some file object self._path2confmods[None] = self.getconftestmodules(anchor) diff -crN pypy-pypy-4a38b43757e3/pypy/conftest.py pypy-pypy-4a38b43757e3.bookaa/pypy/conftest.py *** pypy-pypy-4a38b43757e3/pypy/conftest.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/conftest.py Sat May 26 20:59:51 2012 *************** *** 33,38 **** --- 33,42 ---- raise ValueError("%s not in %s" % (value, PLATFORMS)) set_platform(value, None) + def _set_compiler(opt, opt_str, value, parser): #add by bookaa + from pypy.translator.platform import set_platform + set_platform('host', value) + def pytest_addoption(parser): group = parser.getgroup("pypy options") group.addoption('--view', action="store_true", dest="view", default=False, *************** *** 46,51 **** --- 50,58 ---- group.addoption('-P', '--platform', action="callback", type="string", default="host", callback=_set_platform, help="set up tests to use specified platform as compile/run target") + group.addoption('--cc', action="callback", type="string", #add by bookaa + default="host", callback=_set_compiler, + help="set up tests to use specified compiler") group = parser.getgroup("JIT options") group.addoption('--viewloops', action="store_true", default=False, dest="viewloops", diff -crN pypy-pypy-4a38b43757e3/pypy/rlib/rposix.py pypy-pypy-4a38b43757e3.bookaa/pypy/rlib/rposix.py *** pypy-pypy-4a38b43757e3/pypy/rlib/rposix.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/rlib/rposix.py Sat May 26 20:51:13 2012 *************** *** 24,34 **** separate_module_sources =[''' /* Lifted completely from CPython 3.3 Modules/posix_module.c */ #include /* for _msize */ typedef struct { intptr_t osfhnd; char osfile; } my_ioinfo; ! extern __declspec(dllimport) char * __pioinfo[]; #define IOINFO_L2E 5 #define IOINFO_ARRAY_ELTS (1 << IOINFO_L2E) #define IOINFO_ARRAYS 64 --- 24,35 ---- separate_module_sources =[''' /* Lifted completely from CPython 3.3 Modules/posix_module.c */ #include /* for _msize */ + #include typedef struct { intptr_t osfhnd; char osfile; } my_ioinfo; ! //extern __declspec(dllimport) char * __pioinfo[]; #define IOINFO_L2E 5 #define IOINFO_ARRAY_ELTS (1 << IOINFO_L2E) #define IOINFO_ARRAYS 64 *************** *** 36,41 **** --- 37,58 ---- #define FOPEN 0x01 #define _NO_CONSOLE_FILENO (intptr_t)-2 + char** Get_pioinfo(void** pp) + { + HMODULE hm; + hm = GetModuleHandleA("msvcr90.dll"); + if (hm == 0) + hm = GetModuleHandleA("msvcr90d.dll"); + if (hm == 0) + hm = GetModuleHandleA("msvcrt.dll"); + if (hm) + { + void * p = GetProcAddress(hm, "__pioinfo"); + *pp = GetProcAddress(hm, "_msize"); + return (char**)p; + } + return 0; + } /* This function emulates what the windows CRT does to validate file handles */ int *************** *** 45,56 **** const int i2 = fd & ((1 << IOINFO_L2E) - 1); static size_t sizeof_ioinfo = 0; /* Determine the actual size of the ioinfo structure, * as used by the CRT loaded in memory */ if (sizeof_ioinfo == 0 && __pioinfo[0] != NULL) { ! sizeof_ioinfo = _msize(__pioinfo[0]) / IOINFO_ARRAY_ELTS; } if (sizeof_ioinfo == 0) { /* This should not happen... */ --- 62,77 ---- const int i2 = fd & ((1 << IOINFO_L2E) - 1); static size_t sizeof_ioinfo = 0; + int (*my_msize)(char*) = 0; + char** __pioinfo = Get_pioinfo((void**)&my_msize); + if (__pioinfo == 0) + return 0; /* Determine the actual size of the ioinfo structure, * as used by the CRT loaded in memory */ if (sizeof_ioinfo == 0 && __pioinfo[0] != NULL) { ! sizeof_ioinfo = my_msize(__pioinfo[0]) / IOINFO_ARRAY_ELTS; } if (sizeof_ioinfo == 0) { /* This should not happen... */ *************** *** 75,80 **** --- 96,105 ---- return 0; } ''',] + from pypy.translator.platform import platform + if platform.name == 'mingw32': + separate_module_sources[0] = '''#include /* for intptr_t */ + ''' + separate_module_sources[0] export_symbols = ['_PyVerify_fd'] else: separate_module_sources = [] diff -crN pypy-pypy-4a38b43757e3/pypy/rpython/tool/rfficache.py pypy-pypy-4a38b43757e3.bookaa/pypy/rpython/tool/rfficache.py *** pypy-pypy-4a38b43757e3/pypy/rpython/tool/rfficache.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/rpython/tool/rfficache.py Sat May 26 20:47:42 2012 *************** *** 13,19 **** from pypy.tool.gcc_cache import build_executable_cache def ask_gcc(question, add_source=""): ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h'] if os.name != 'nt': includes += ['inttypes.h'] include_string = "\n".join(["#include <%s>" % i for i in includes]) --- 13,23 ---- from pypy.tool.gcc_cache import build_executable_cache def ask_gcc(question, add_source=""): ! from pypy.translator.platform import platform ! if platform.name == 'mingw32': ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h', 'stdint.h'] ! else: ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h'] if os.name != 'nt': includes += ['inttypes.h'] include_string = "\n".join(["#include <%s>" % i for i in includes]) diff -crN pypy-pypy-4a38b43757e3/pypy/translator/platform/posix.py pypy-pypy-4a38b43757e3.bookaa/pypy/translator/platform/posix.py *** pypy-pypy-4a38b43757e3/pypy/translator/platform/posix.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/translator/platform/posix.py Sat May 26 20:47:42 2012 *************** *** 55,60 **** --- 55,62 ---- if relto: response_file = relto.bestrelpath(response_file) + if self.name == 'mingw32': + return ["-Wl,--export-all-symbols,--version-script=%s" % (response_file,)] return ["-Wl,--export-dynamic,--version-script=%s" % (response_file,)] def _link(self, cc, ofiles, link_args, standalone, exe_name): -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bookaa26.dif Type: application/octet-stream Size: 7411 bytes Desc: not available URL: From tbaldridge at gmail.com Sat May 26 22:19:38 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Sat, 26 May 2012 15:19:38 -0500 Subject: [pypy-dev] Can someone review my RPython code? Message-ID: So here's the current status of my Polymorphic functions in as a pypy module: https://bitbucket.org/halgari/pypy/src/d9e7191d13df/pypy/module/clojure/functions.py I have a question about the way "space" is handled. My problem is this, I'd like to create several instances of PolymorphicFn on the interpreter level, so I can register some types with the function when the module is loaded. However, I'm not sure how I should do this. If I don't implement __new__ pypy complains that "using object.__new__ is dangerous, use MyClass.__new__ instead". However, if I implement __new__ I have to pass a space object in. Anyway, I'm sorry if I'm not making much sense here, but I guess I'm not too clear on how to create "global" objects. For instance, given my above functions.py file, how do I do this: first = PolymorphicFn() first.extend(space.type(space.wrap(None)), space.wrap(lambda w_x: None)) first.extend(Cons.typedef, Cons.typedef.rawdict("first")) To do this, I need an instance of space, but how do I get this in a global (module level) context? Thanks for any help. Timothy From kwa at kuwata-lab.com Sun May 27 06:24:39 2012 From: kwa at kuwata-lab.com (Makoto Kuwata) Date: Sun, 27 May 2012 13:24:39 +0900 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 Message-ID: Hi, I found that sys._getframe() (or sys._getframe().f_locals) is very slow in PyPy 1.8. For example, the following takes only 0.0398 sec on PyPy 1.8: def f5(x): return f4(x) def f4(x): return f3(x) def f2(x): return f1(x) def f1(x): reutrn None ## just return None for _ in xrange(1000000): f5(0) But the next one takes 1.3563 sec. def f5(x): return f4(x) def f4(x): return f3(x) def f2(x): return f1(x) def f1(x): reutrn sys._getframe(1) ## calls sys._getframe(1) for _ in xrange(1000000): f5(0) Is there any plan to make it faster in the future release? Background ---------- I'm an author of Tenjin, a high-speed template engine for Python. Tenjin supports PyPy officially, and I received a report that there are some cases in which Tenjin is very slow on PyPy. I investigated report and found that it is the sys._getframe() and sys._getframe().f_locals that makes Tenjin much slow on PyPy. Tenjin uses sys._getframe(1).f_locals in order to access to caller's local variables. It is necessary for Tenjin. Benchmark --------- Here is a benchmark script to measure cost of sys._getframe(1) and sys._getframe(1).f_locals. (You must install Benchmarker library to run it.) https://gist.github.com/2802160 Benchmark result shows that: * Cost of sys._getframe(1) and sys._getframe(1).f_locals is not so much on CPython but very big on PyPy. * Nested depth os function call affects to cost of them. For example: def f5(x): return f4(x) def f4(x): return f3(x) def f2(x): return f1(x) def f1(x): reutrn sys._getframe(1) f5(0) is much slower than: def f2(x): return f1(x) def f1(x): reutrn sys._getframe(1) f2(0) By the way, PyPy is awesome. It is not only faster than CPython but also has high-level compatibility with CPython. Almost of all test script of Tenjin works on PyPy. Great. I hope Tenjin works much faster on PyPy. -- regards, makoto kuwata From Ronny.Pfannschmidt at gmx.de Sun May 27 11:09:38 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sun, 27 May 2012 11:09:38 +0200 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: Message-ID: <4FC1EF52.9010909@gmx.de> Hi Makoto, currently sys._getframe() is mostly killing the jit since it currently needs access to all frame objects, if i remember correct (arigo, fijal or anto will know for sure), the jit currently just bails out if it gets a sys._getframe call (which easyly explain the ~34 times slower in your test) i think there are ways to support it, but i currently don't have the time/knowledge to check which of them work -- Ronny On 05/27/2012 06:24 AM, Makoto Kuwata wrote: > Hi, > > I found that sys._getframe() (or sys._getframe().f_locals) is > very slow in PyPy 1.8. > For example, the following takes only 0.0398 sec on PyPy 1.8: > > def f5(x): return f4(x) > def f4(x): return f3(x) > def f2(x): return f1(x) > def f1(x): reutrn None ## just return None > for _ in xrange(1000000): > f5(0) > > But the next one takes 1.3563 sec. > > def f5(x): return f4(x) > def f4(x): return f3(x) > def f2(x): return f1(x) > def f1(x): reutrn sys._getframe(1) ## calls sys._getframe(1) > for _ in xrange(1000000): > f5(0) > > Is there any plan to make it faster in the future release? > > > Background > ---------- > > I'm an author of Tenjin, a high-speed template engine for Python. > Tenjin supports PyPy officially, and I received a report that > there are some cases in which Tenjin is very slow on PyPy. > > I investigated report and found that it is the sys._getframe() > and sys._getframe().f_locals that makes Tenjin much slow on PyPy. > > Tenjin uses sys._getframe(1).f_locals in order to access to > caller's local variables. It is necessary for Tenjin. > > > Benchmark > --------- > > Here is a benchmark script to measure cost of sys._getframe(1) > and sys._getframe(1).f_locals. > (You must install Benchmarker library to run it.) > > https://gist.github.com/2802160 > > Benchmark result shows that: > > * Cost of sys._getframe(1) and sys._getframe(1).f_locals is > not so much on CPython but very big on PyPy. > * Nested depth os function call affects to cost of them. > For example: > > def f5(x): return f4(x) > def f4(x): return f3(x) > def f2(x): return f1(x) > def f1(x): reutrn sys._getframe(1) > f5(0) > > is much slower than: > > def f2(x): return f1(x) > def f1(x): reutrn sys._getframe(1) > f2(0) > > > > By the way, PyPy is awesome. > It is not only faster than CPython but also has high-level > compatibility with CPython. Almost of all test script of Tenjin > works on PyPy. Great. > I hope Tenjin works much faster on PyPy. > > -- > regards, > makoto kuwata > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From kwa at kuwata-lab.com Sun May 27 12:30:28 2012 From: kwa at kuwata-lab.com (Makoto Kuwata) Date: Sun, 27 May 2012 19:30:28 +0900 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: <4FC1EF52.9010909@gmx.de> References: <4FC1EF52.9010909@gmx.de> Message-ID: Hi Ronny, Thank you for your reply. You mean that sys._getframe() will remain slow on PyPy. I'll try other way (instead of sys._getframe()) to make Tenjin faster on PyPy. -- regards, makoto kuwata On Sun, May 27, 2012 at 6:09 PM, Ronny Pfannschmidt wrote: > Hi Makoto, > > currently sys._getframe() is mostly killing the jit > since it currently needs access to all frame objects, > > if i remember correct (arigo, fijal or anto will know for sure), > the jit currently just bails out if it gets a sys._getframe call > > (which easyly explain the ~34 times slower ?in your test) > > i think there are ways to support it, > but i currently don't have the time/knowledge to check which of them work > > -- Ronny > > > On 05/27/2012 06:24 AM, Makoto Kuwata wrote: >> >> Hi, >> >> I found that sys._getframe() (or sys._getframe().f_locals) is >> very slow in PyPy 1.8. >> For example, the following takes only 0.0398 sec on PyPy 1.8: >> >> ? ? ? def f5(x): return f4(x) >> ? ? ? def f4(x): return f3(x) >> ? ? ? def f2(x): return f1(x) >> ? ? ? def f1(x): reutrn None ? ? ? ? ? ? ? ## just return None >> ? ? ? for _ in xrange(1000000): >> ? ? ? ? ? f5(0) >> >> But the next one takes 1.3563 sec. >> >> ? ? ? def f5(x): return f4(x) >> ? ? ? def f4(x): return f3(x) >> ? ? ? def f2(x): return f1(x) >> ? ? ? def f1(x): reutrn sys._getframe(1) ? ?## calls sys._getframe(1) >> ? ? ? for _ in xrange(1000000): >> ? ? ? ? ? f5(0) >> >> Is there any plan to make it faster in the future release? >> >> >> Background >> ---------- >> >> I'm an author of Tenjin, a high-speed template engine for Python. >> Tenjin supports PyPy officially, and I received a report that >> there are some cases in which Tenjin is very slow on PyPy. >> >> I investigated report and found that it is the sys._getframe() >> and sys._getframe().f_locals that makes Tenjin much slow on PyPy. >> >> Tenjin uses sys._getframe(1).f_locals in order to access to >> caller's local variables. It is necessary for Tenjin. >> >> >> Benchmark >> --------- >> >> Here is a benchmark script to measure cost of sys._getframe(1) >> and sys._getframe(1).f_locals. >> (You must install Benchmarker library to run it.) >> >> https://gist.github.com/2802160 >> >> Benchmark result shows that: >> >> * Cost of sys._getframe(1) and sys._getframe(1).f_locals is >> ? not so much on CPython but very big on PyPy. >> * Nested depth os function call affects to cost of them. >> ? For example: >> >> ? ? ? def f5(x): return f4(x) >> ? ? ? def f4(x): return f3(x) >> ? ? ? def f2(x): return f1(x) >> ? ? ? def f1(x): reutrn sys._getframe(1) >> ? ? ? f5(0) >> >> ? is much slower than: >> >> ? ? ? def f2(x): return f1(x) >> ? ? ? def f1(x): reutrn sys._getframe(1) >> ? ? ? f2(0) >> >> >> >> By the way, PyPy is awesome. >> It is not only faster than CPython but also has high-level >> compatibility with CPython. Almost of all test script of Tenjin >> works on PyPy. Great. >> I hope Tenjin works much faster on PyPy. >> >> -- >> regards, >> makoto kuwata >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > From fijall at gmail.com Sun May 27 12:45:35 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 27 May 2012 12:45:35 +0200 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: On Sun, May 27, 2012 at 12:30 PM, Makoto Kuwata wrote: > Hi Ronny, > > Thank you for your reply. > You mean that sys._getframe() will remain slow on PyPy. > I'll try other way (instead of sys._getframe()) to make Tenjin faster on > PyPy. > Hi Makoto. sys._getframe(1).f_locals will stay slow. The reason for this is that you have to create all the locals and put them in a dictionary (they normally don't even exist on the heap). Because of that we decided the JIT should simply bail out and give up trying to optimize this particular code. Note that as documented on the python website, this functionality is mostly for implementing debuggers and such (where speed does not matter), do you *really* need your callers locals? Sounds like a very deep magic to me, I can point you to this presentation [1] as to why it might be a bad idea. [1].. http://www.youtube.com/watch?v=8e0l_Dt28MQ Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob at openend.se Sun May 27 12:49:12 2012 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Sun, 27 May 2012 12:49:12 +0200 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: <201205271249.13278.jacob@openend.se> One of the ways the JIT makes things go faster is by not generating any frames at all. When you ask for the contents of a frame, a virtual frame is generated, to provide the information you are asking for. This takes a fair bit of time, making your code slow. Jacob Hall?n Sunday 27 May 2012 you wrote: > Hi Ronny, > > Thank you for your reply. > You mean that sys._getframe() will remain slow on PyPy. > I'll try other way (instead of sys._getframe()) to make Tenjin faster on > PyPy. > > -- > regards, > makoto kuwata > > On Sun, May 27, 2012 at 6:09 PM, Ronny Pfannschmidt > > wrote: > > Hi Makoto, > > > > currently sys._getframe() is mostly killing the jit > > since it currently needs access to all frame objects, > > > > if i remember correct (arigo, fijal or anto will know for sure), > > the jit currently just bails out if it gets a sys._getframe call > > > > (which easyly explain the ~34 times slower in your test) > > > > i think there are ways to support it, > > but i currently don't have the time/knowledge to check which of them work > > > > -- Ronny > > > > On 05/27/2012 06:24 AM, Makoto Kuwata wrote: > >> Hi, > >> > >> I found that sys._getframe() (or sys._getframe().f_locals) is > >> very slow in PyPy 1.8. > >> For example, the following takes only 0.0398 sec on PyPy 1.8: > >> > >> def f5(x): return f4(x) > >> def f4(x): return f3(x) > >> def f2(x): return f1(x) > >> def f1(x): reutrn None ## just return None > >> for _ in xrange(1000000): > >> f5(0) > >> > >> But the next one takes 1.3563 sec. > >> > >> def f5(x): return f4(x) > >> def f4(x): return f3(x) > >> def f2(x): return f1(x) > >> def f1(x): reutrn sys._getframe(1) ## calls sys._getframe(1) > >> for _ in xrange(1000000): > >> f5(0) > >> > >> Is there any plan to make it faster in the future release? > >> > >> > >> Background > >> ---------- > >> > >> I'm an author of Tenjin, a high-speed template engine for Python. > >> Tenjin supports PyPy officially, and I received a report that > >> there are some cases in which Tenjin is very slow on PyPy. > >> > >> I investigated report and found that it is the sys._getframe() > >> and sys._getframe().f_locals that makes Tenjin much slow on PyPy. > >> > >> Tenjin uses sys._getframe(1).f_locals in order to access to > >> caller's local variables. It is necessary for Tenjin. > >> > >> > >> Benchmark > >> --------- > >> > >> Here is a benchmark script to measure cost of sys._getframe(1) > >> and sys._getframe(1).f_locals. > >> (You must install Benchmarker library to run it.) > >> > >> https://gist.github.com/2802160 > >> > >> Benchmark result shows that: > >> > >> * Cost of sys._getframe(1) and sys._getframe(1).f_locals is > >> not so much on CPython but very big on PyPy. > >> * Nested depth os function call affects to cost of them. > >> For example: > >> > >> def f5(x): return f4(x) > >> def f4(x): return f3(x) > >> def f2(x): return f1(x) > >> def f1(x): reutrn sys._getframe(1) > >> f5(0) > >> > >> is much slower than: > >> > >> def f2(x): return f1(x) > >> def f1(x): reutrn sys._getframe(1) > >> f2(0) > >> > >> > >> > >> By the way, PyPy is awesome. > >> It is not only faster than CPython but also has high-level > >> compatibility with CPython. Almost of all test script of Tenjin > >> works on PyPy. Great. > >> I hope Tenjin works much faster on PyPy. > >> > >> -- > >> regards, > >> makoto kuwata > >> _______________________________________________ > >> pypy-dev mailing list > >> pypy-dev at python.org > >> http://mail.python.org/mailman/listinfo/pypy-dev > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From newsletter at ensinfo.org Sun May 27 13:53:15 2012 From: newsletter at ensinfo.org (ENS Newsletter) Date: Sun, 27 May 2012 13:53:15 +0200 Subject: [pypy-dev] Unsubscribe Newsletter ENS Message-ID: An HTML attachment was scrubbed... URL: From kwa at kuwata-lab.com Sun May 27 14:27:59 2012 From: kwa at kuwata-lab.com (Makoto Kuwata) Date: Sun, 27 May 2012 21:27:59 +0900 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: Hi Maciej, On Sun, May 27, 2012 at 7:45 PM, Maciej Fijalkowski wrote: > Hi Makoto. > > sys._getframe(1).f_locals will stay slow. The reason for this is that you > have to create all the locals and put them in a dictionary (they normally > don't even exist on the heap). Because of that we decided the JIT should > simply bail out and give up trying to optimize this particular code. I agree to PyPy team's decision. Some points may be slower on PyPy, but almost of all is much faster than CPython. > Note > that as documented on the python website, this functionality is mostly for > implementing debuggers and such (where speed does not matter), do you > *really* need your callers locals? Yes. I need to access to caller's local variables in my product (= Tenjin template engine) and sys._getframe() is reasonable solution for it (at least in CPython). > Sounds like a very deep magic to me, I > can point you to this presentation [1] as to why it might be a bad idea. > > [1]..?http://www.youtube.com/watch?v=8e0l_Dt28MQ I know that sys._getframe() is kind of black magic, but I must use it because it is only the way to access to caller's local variables. My goal is to access to caller's local variables, and sys._getframe() is just a way to reach to goal. -- regards, makoto kuwata From fijall at gmail.com Sun May 27 14:31:10 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 27 May 2012 14:31:10 +0200 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: On Sun, May 27, 2012 at 2:27 PM, Makoto Kuwata wrote: > Hi Maciej, > > On Sun, May 27, 2012 at 7:45 PM, Maciej Fijalkowski > wrote: > > Hi Makoto. > > > > sys._getframe(1).f_locals will stay slow. The reason for this is that you > > have to create all the locals and put them in a dictionary (they normally > > don't even exist on the heap). Because of that we decided the JIT should > > simply bail out and give up trying to optimize this particular code. > > I agree to PyPy team's decision. > Some points may be slower on PyPy, but almost of all is much faster > than CPython. > > > > Note > > that as documented on the python website, this functionality is mostly > for > > implementing debuggers and such (where speed does not matter), do you > > *really* need your callers locals? > > Yes. I need to access to caller's local variables in my product (= > Tenjin template engine) > and sys._getframe() is reasonable solution for it (at least in CPython). > > > > Sounds like a very deep magic to me, I > > can point you to this presentation [1] as to why it might be a bad idea. > > > > [1].. http://www.youtube.com/watch?v=8e0l_Dt28MQ > > I know that sys._getframe() is kind of black magic, but I must use it > because it is only the way to access to caller's local variables. > > My goal is to access to caller's local variables, and sys._getframe() is > just a way to reach to goal. > My point is mostly that style-wise the design decision that led you to "I need to inspect the callers variables" was probably not a very good one. It's very tempting to use black magic, because writing explicit passing of arguments is too verbose, but remember that explicit is always better than implicit. I won't try to convince you any more - it's more about style than about what can or cannot be made fast on pypy. Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Sun May 27 14:35:35 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 27 May 2012 08:35:35 -0400 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: On Sun, May 27, 2012 at 8:31 AM, Maciej Fijalkowski wrote: > On Sun, May 27, 2012 at 2:27 PM, Makoto Kuwata wrote: > >> Hi Maciej, >> >> On Sun, May 27, 2012 at 7:45 PM, Maciej Fijalkowski >> wrote: >> > Hi Makoto. >> > >> > sys._getframe(1).f_locals will stay slow. The reason for this is that >> you >> > have to create all the locals and put them in a dictionary (they >> normally >> > don't even exist on the heap). Because of that we decided the JIT should >> > simply bail out and give up trying to optimize this particular code. >> >> I agree to PyPy team's decision. >> Some points may be slower on PyPy, but almost of all is much faster >> than CPython. >> >> >> > Note >> > that as documented on the python website, this functionality is mostly >> for >> > implementing debuggers and such (where speed does not matter), do you >> > *really* need your callers locals? >> >> Yes. I need to access to caller's local variables in my product (= >> Tenjin template engine) >> and sys._getframe() is reasonable solution for it (at least in CPython). >> >> >> > Sounds like a very deep magic to me, I >> > can point you to this presentation [1] as to why it might be a bad idea. >> > >> > [1].. http://www.youtube.com/watch?v=8e0l_Dt28MQ >> >> I know that sys._getframe() is kind of black magic, but I must use it >> because it is only the way to access to caller's local variables. >> >> My goal is to access to caller's local variables, and sys._getframe() is >> just a way to reach to goal. >> > > My point is mostly that style-wise the design decision that led you to "I > need to inspect the callers variables" was probably not a very good one. > It's very tempting to use black magic, because writing explicit passing of > arguments is too verbose, but remember that explicit is always better than > implicit. I won't try to convince you any more - it's more about style than > about what can or cannot be made fast on pypy. > > Cheers, > fijal > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > If this is a template engine there's no reason you have to implement scope using python's scope directly, both Django, Jinja, and others implement a different scoping system on top of Pythons. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwa at kuwata-lab.com Sun May 27 14:40:10 2012 From: kwa at kuwata-lab.com (Makoto Kuwata) Date: Sun, 27 May 2012 21:40:10 +0900 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: On Sun, May 27, 2012 at 9:31 PM, Maciej Fijalkowski wrote: > My point is mostly that style-wise the design decision that led you to "I > need to inspect the callers variables" was probably not a very good one. > It's very tempting to use black magic, because writing explicit passing of > arguments is too verbose, but remember that explicit is always better than > implicit. I won't try to convince you any more - it's more about style than > about what can or cannot be made fast on pypy. Main purpose to access to caller's local variables is to hide implementation details of template engine from users. 'Explicit vs. Implicit' is not the point at all. -- regards, makoto kuwata From kwa at kuwata-lab.com Sun May 27 14:53:18 2012 From: kwa at kuwata-lab.com (Makoto Kuwata) Date: Sun, 27 May 2012 21:53:18 +0900 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: On Sun, May 27, 2012 at 9:35 PM, Alex Gaynor wrote: > > If this is a template engine there's no reason you have to implement scope > using python's scope directly, both Django, Jinja, and others implement a > different scoping system on top of Pythons. > Of course it is possible, and that is one of the reason why other template engines (such as Django or Jinja2) are very fat and slower than my Tenjin. There are some approaches to implement template engine. My approach is different from Djang or Jinja2. It's that easy. I must say again: I agree to PyPy dev team's decision and I'll try other approach to make my product faster on PyPy as well as CPython. I don't have any intention to blame PyPy. It's awesome software. -- regards, makoto kuwata From arigo at tunes.org Sun May 27 17:38:36 2012 From: arigo at tunes.org (Armin Rigo) Date: Sun, 27 May 2012 17:38:36 +0200 Subject: [pypy-dev] Differences between app and interp level calling methods In-Reply-To: References: Message-ID: Hi Timothy, On Fri, May 25, 2012 at 10:48 PM, Timothy Baldridge wrote: > For something like this, am I going to see much of a performance boost > by dropping to the interpreter level from the dispatch between these > three sub-function types? I can't answer this question, because it is too vague. It depends on the rest of the code, and on what you need to do exactly in the mentioned logic. Then it depends if you want the performance before the JIT is applied, or after. Finally it depends on whether the JIT is fully correctly applied or not (and this includes whether the JIT can really be applied in your case or we need refactorings of either your code or the JIT itself). You have to either try it out and measure alternatives for yourself, or in order for us to come up with a guess, we'd need to investigate your whole source. A bient?t, Armin. From naveen.garg at gmail.com Sun May 27 17:39:52 2012 From: naveen.garg at gmail.com (Naveen Garg) Date: Sun, 27 May 2012 10:39:52 -0500 Subject: [pypy-dev] checkpointing pypy sandbox in dmtcp Message-ID: I couldn't figure out what antonio cuni was trying to do with the dmtcp python api here: https://bitbucket.org/antocuni/misc/src/73621b70935f/hack/checkpointing/dmtcp_sum.py anyways, Running the pypy interpreter in dmtcp seems to work out of the box. This works even with the sandboxed interpreter. I did: 1. translate the sandbox using instructions here: http://doc.pypy.org/en/latest/sandbox.html 2. mv pypy-c ~/bin/pypy/bin/pypy-c-sandbox bash1> dmtcp_coordinator bash2> dmtcp_checkpoint python pypy_interact.py --tmp=myexported ~/bin/pypy/bin/pypy-c-sandbox bash1> c # save checkpoint bash2> -c # crash pypy sandbox bash2> dmtcp_restart_script.sh # revive pypy sandbox :) profit -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sun May 27 17:44:08 2012 From: arigo at tunes.org (Armin Rigo) Date: Sun, 27 May 2012 17:44:08 +0200 Subject: [pypy-dev] add --cc option to pytest In-Reply-To: <68622C6A999E45D0A12075506A6EB2A2@vSHliutaotao> References: <68622C6A999E45D0A12075506A6EB2A2@vSHliutaotao> Message-ID: Hi Bookaa, On Sat, May 26, 2012 at 3:55 AM, bookaa wrote: > ?pytest.py --cc=mingw32-gcc > pypy/translator/test/test_exceptiontransform.py::TestOOType::()::test_simple I didn't know this syntax. The official syntax (as far as I'm concerned) is: pytest.py filetest.py -k test_simple > E?????? e:\tem\usession-default-137\main.il(161) : error : syntax error at > token '[' in:???? call [mscorlib]System.Object rpyexc_fetch_value() Sorry, can't help here. I know next to nothing about CLI/.NET. A bient?t, Armin. From arigo at tunes.org Sun May 27 17:54:47 2012 From: arigo at tunes.org (Armin Rigo) Date: Sun, 27 May 2012 17:54:47 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: <201205230429.q4N4TebX003789@theraft.openend.se> References: <201205230429.q4N4TebX003789@theraft.openend.se> Message-ID: Hi Laura, On Wed, May 23, 2012 at 6:29 AM, Laura Creighton wrote: > tons of things that I agree with more or less wholesale, though I think > that there is a slightly larger market for Cython in the immediate > future ... Sure, I never meant to give any market size estimate. > Sounds great to me. ?Except that Mike Pell (LuaJit's author) hasn't > been cc'd ?on this note. He has shown up in #pypy to discuss it. >?Because, dear God, > if he had not already existed and made this thing, then *we* > would be the people with the best working FFI. I want to emphasis here that it's done "the LuaJIT way". I don't want to pretend having invented anything myself. If it were not for LuaJIT, then very probably we would not have found this solution --- although it seems very natural in hindsight. > No NO NO -- our way was an experiment and too messy to be used. Unsure what you mean here... I've used the Python ffi interface on some examples, and it seems both clean enough and much more natural than packaging tons of objects that you have to learn about first. It is maybe as close to a no-API interface that you can get, provided you already know Python and have a basic knowledge of C. Example: http://bpaste.net/show/30171/ This is as close to the original C code as possible, with a small page of C declarations copy-pasted from the man pages. I think that this example alone shows that the approach really works, although it should be tried on larger examples too. A bient?t, Armin. From arigo at tunes.org Sun May 27 18:43:09 2012 From: arigo at tunes.org (Armin Rigo) Date: Sun, 27 May 2012 18:43:09 +0200 Subject: [pypy-dev] Can someone review my RPython code? In-Reply-To: References: Message-ID: Hi, On Sat, May 26, 2012 at 10:19 PM, Timothy Baldridge wrote: > To do this, I need an instance of space, but how do I get this in a > global (module level) context? Try to move the initialization of this data outside module globals. See for example pypy/module/sys/state.py, where exactly one prebuilt instance of State is created during translation by the call "space.fromcache(State)". In State.__init__() you can do anything, as it runs during translation and is not itself translated. A bient?t, Armin. From matti.picus at gmail.com Sun May 27 19:38:40 2012 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 27 May 2012 20:38:40 +0300 Subject: [pypy-dev] add --cc option to pytest In-Reply-To: <68622C6A999E45D0A12075506A6EB2A2@vSHliutaotao> References: <68622C6A999E45D0A12075506A6EB2A2@vSHliutaotao> Message-ID: <4FC266A0.4050607@gmail.com> An HTML attachment was scrubbed... URL: From gelonida at gmail.com Sun May 27 18:56:41 2012 From: gelonida at gmail.com (Gelonida N) Date: Sun, 27 May 2012 18:56:41 +0200 Subject: [pypy-dev] how to build pypy with psycopg (links / requirements) Message-ID: Hi, Is there any good document / tutorial showing how to build pypy with the Rpython version of psycopg? Thanks an advance for good links. I'm working on Ubuntu 12.04 From matti.picus at gmail.com Sun May 27 19:57:53 2012 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 27 May 2012 20:57:53 +0300 Subject: [pypy-dev] pytest test_rposix.py with mingw32 In-Reply-To: <5E5F88FFB39147BB8A38143E94DA595F@vSHliutaotao> References: <5E5F88FFB39147BB8A38143E94DA595F@vSHliutaotao> Message-ID: <4FC26B21.9010901@gmail.com> On 26/05/2012 4:47 PM, bookaa wrote: > keywords: _PyVerify_fd __pioinfo _msize > PyPy can not pass this test: > pytest.py --cc=mingw32-gcc > pypy\rlib\test\test_rposix.py::TestPosixUnicode::()::test_is_valid_fd > After 2 days of hard work, I finally realise why. Patch file attached. > The problem is in _PyVerify_fd: > FILE* f = fopen("d:\\55.txt","w"); > int no = fileno(f); > int flg1 = _PyVerify_fd(no); > fclose(f); > int flg2 = _PyVerify_fd(no); > printf("flg1 %d should be 1, flg2 %d should be 0", flg1, flg2) > in file rposix.py, source of function _PyVerify_fd is write to a c > file, and will be compile to a Windows > DLL file. In this function, it use > extern __declspec(dllimport) char * __pioinfo[]; > __pioinfo as a global variable. > I find use __pioinfo in a DLL file is not safe. the __pioinfo and > _msize may export from many DLL: > msvcrt.dll > msvcr90.dll > msvcr90d.dll > or maybe msvcr70.dll ... > If we fopen and fileno in a EXE file, and call _PyVerify_fd in a DLL > file, its very danger. The EXE > and DLL may reference different __pioinfo and get error. > So I think test_rposix.py::TestPosixUnicode::()::test_is_valid_fd can > pass in MSVC is only a coincidence. > _PyVerify_fd in a DLL and use dllimport __pioinfo is not safe. > In my fix, I do not assume any DLL we should use. I try to find the > current DLL which export __pioinfo already > loaded. Sure this is not very good. But I think if we must place > _PyVerify_fd in a DLL, no good solution. > After this path, this tests are pass: > pytest.py --cc=mingw32-gcc pypy\rlib\test\test_rposix.py > pytest.py pypy\rlib\test\test_rposix.py Thanks for your help tracking this down. It's nice to see someone else who is interested in the windows port. Most of this patch was already merged into the default branch a few days ago. Also: it seems like an issue during testing only. The patch should somehow reflect this without hardcoding runtime info into the translated pypy. Also please do not use --cc=mingw32 syntax for tests. If you set an environment variable to CC: export CC=mingw32 (in a unix-like shell) set CC=mingw32 (in the cmd shell) it will use a compiler named mingw32 Matti From fijall at gmail.com Sun May 27 20:01:22 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 27 May 2012 20:01:22 +0200 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: On Sun, May 27, 2012 at 2:53 PM, Makoto Kuwata wrote: > On Sun, May 27, 2012 at 9:35 PM, Alex Gaynor > wrote: > > > > If this is a template engine there's no reason you have to implement > scope > > using python's scope directly, both Django, Jinja, and others implement a > > different scoping system on top of Pythons. > > > > Of course it is possible, and that is one of the reason why other > template engines > (such as Django or Jinja2) are very fat and slower than my Tenjin. > There are some approaches to implement template engine. > My approach is different from Djang or Jinja2. It's that easy. > I seriously have to say [citation needed]. Do you have benchmark figures? I'm especially interested in jinja2 and spitfire (spitfire is quite fast). > > I must say again: I agree to PyPy dev team's decision and I'll try other > approach to make my product faster on PyPy as well as CPython. > I don't have any intention to blame PyPy. It's awesome software. > > -- > regards, > makoto kuwata > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Sun May 27 21:33:46 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sun, 27 May 2012 21:33:46 +0200 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: Hi, 2012/5/27 Makoto Kuwata > I must say again: I agree to PyPy dev team's decision and I'll try other > approach to make my product faster on PyPy as well as CPython. > I don't have any intention to blame PyPy. It's awesome software. > It's not a "team's decision"; PyPy is faster because most of the time, it avoids building the dictionary of local variables, and sometimes not allocate local objects at all. Or said the other way round: CPython is slower because it always builds and maintains these objects. By accessing f_locals, your code explicitly requires this dictionary to be built, which effectively voids all the optimizations above. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwa at kuwata-lab.com Sun May 27 23:43:04 2012 From: kwa at kuwata-lab.com (Makoto Kuwata) Date: Mon, 28 May 2012 06:43:04 +0900 Subject: [pypy-dev] sys._getframe(1).f_locals is very slow on PyPy 1.8 In-Reply-To: References: <4FC1EF52.9010909@gmx.de> Message-ID: On Mon, May 28, 2012 at 3:01 AM, Maciej Fijalkowski wrote: > On Sun, May 27, 2012 at 2:53 PM, Makoto Kuwata wrote: >> >> On Sun, May 27, 2012 at 9:35 PM, Alex Gaynor >> wrote: >> > >> > If this is a template engine there's no reason you have to implement >> > scope >> > using python's scope directly, both Django, Jinja, and others implement >> > a >> > different scoping system on top of Pythons. >> > >> >> Of course it is possible, and that is one of the reason why other >> template engines >> (such as Django or Jinja2) are very fat and slower than my Tenjin. >> There are some approaches to implement template engine. >> My approach is different from Djang or Jinja2. It's that easy. > > > I seriously have to say [citation needed]. Do you have benchmark figures? > I'm especially interested in jinja2 and spitfire (spitfire is quite fast). Template engine benchmarking is not the point of current discussion, but if you are interested in it, try: $ wget http://pypi.python.org/packages/source/T/Tenjin/Tenjin-1.1.1.tar.gz $ tar xf Tenjin-1.1.1.tar.gz $ cd Tenjin-1.1.1/benchmark $ easy_install Tenjin Mako Jinja2 Django Templetor Cheetah Genshi Kid $ python bench.py -h $ python bench.py -q -n 10000 Here is an example of benchmark result (on CPython): http://www.kuwata-lab.com/tenjin/pytenjin-users-guide.html#benchmark http://www.kuwata-lab.com/tenjin/ http://www.slideshare.net/kwatch/how-to-create-a-highspeed-template-engine-in-python (page 5) This shows that Tenjin is: * twice faster than Mako or Jinja2 * twenty times faster than Django (Sorry I don't know Spitfire.) -- regards, makoto kuwata From kostia.lopuhin at gmail.com Mon May 28 17:03:23 2012 From: kostia.lopuhin at gmail.com (=?KOI8-R?B?68/T1NEg7M/Q1cjJzg==?=) Date: Mon, 28 May 2012 19:03:23 +0400 Subject: [pypy-dev] how to build pypy with psycopg (links / requirements) In-Reply-To: References: Message-ID: If there is no such document, I am willing to try to make psycopg work with pypy, if it is a suitable work for pypy newbie, and if someone can give me some hints. Is it reasonable to start with building pypy with psycopg support from here https://bitbucket.org/alex_gaynor/pypy-postgresql and then try to merge from upstream? 2012/5/27 Gelonida N : > Hi, > > Is there any good document / tutorial showing how to build pypy with the > Rpython version of psycopg? > > Thanks an advance for good links. > > I'm working on Ubuntu 12.04 > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From rorsoft at gmail.com Mon May 28 17:04:18 2012 From: rorsoft at gmail.com (bookaa) Date: Mon, 28 May 2012 23:04:18 +0800 Subject: [pypy-dev] test_rpoll.py::test_translate with MinGW32 Message-ID: with set cc=mingw32-gcc this test fail: pytest.py pypy/rlib/test/test_rpoll.py::test_translate this is how to patch: diff -crN pypy-pypy-4a38b43757e3/pypy/rlib/_rsocket_rffi.py pypy-pypy-4a38b43757e3.bookaa/pypy/rlib/_rsocket_rffi.py *** pypy-pypy-4a38b43757e3/pypy/rlib/_rsocket_rffi.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/rlib/_rsocket_rffi.py Mon May 28 22:00:03 2012 *************** *** 58,69 **** header_lines = [ '#include ', '#include ', - '#include ', # winsock2 defines AF_UNIX, but not sockaddr_un '#undef AF_UNIX', ] if _MSVC: header_lines.extend([ # these types do not exist on microsoft compilers 'typedef int ssize_t;', 'typedef unsigned __int16 uint16_t;', --- 58,69 ---- header_lines = [ '#include ', '#include ', # winsock2 defines AF_UNIX, but not sockaddr_un '#undef AF_UNIX', ] if _MSVC: header_lines.extend([ + '#include ', # these types do not exist on microsoft compilers 'typedef int ssize_t;', 'typedef unsigned __int16 uint16_t;', *************** *** 71,77 **** ]) else: # MINGW includes = ('stdint.h',) ! """ header_lines.extend([ '''\ #ifndef _WIN32_WINNT --- 71,77 ---- ]) else: # MINGW includes = ('stdint.h',) ! header_lines.extend([ '''\ #ifndef _WIN32_WINNT *************** *** 89,95 **** u_long keepaliveinterval; };''' ]) ! """ HEADER = '\n'.join(header_lines) COND_HEADER = '' constants = {} --- 89,95 ---- u_long keepaliveinterval; };''' ]) ! HEADER = '\n'.join(header_lines) COND_HEADER = '' constants = {} diff -crN pypy-pypy-4a38b43757e3/pypy/rpython/tool/rfficache.py pypy-pypy-4a38b43757e3.bookaa/pypy/rpython/tool/rfficache.py *** pypy-pypy-4a38b43757e3/pypy/rpython/tool/rfficache.py Tue May 22 14:03:20 2012 --- pypy-pypy-4a38b43757e3.bookaa/pypy/rpython/tool/rfficache.py Mon May 28 21:58:02 2012 *************** *** 13,19 **** from pypy.tool.gcc_cache import build_executable_cache def ask_gcc(question, add_source=""): ! includes = ['stdlib.h', 'stdio.h', 'sys/types.h'] if os.name != 'nt': includes += ['inttypes.h'] include_string = "\n".join(["#include -------------- next part -------------- A non-text attachment was scrubbed... Name: bookaa28.dif Type: application/octet-stream Size: 6069 bytes Desc: not available URL: From dynamicgl at gmail.com Mon May 28 17:09:41 2012 From: dynamicgl at gmail.com (gelin yan) Date: Mon, 28 May 2012 23:09:41 +0800 Subject: [pypy-dev] how to build pypy with psycopg (links / requirements) In-Reply-To: References: Message-ID: On Mon, May 28, 2012 at 11:03 PM, ????? ??????? wrote: > If there is no such document, I am willing to try to make psycopg work > with pypy, if it is a suitable work for pypy newbie, and if someone > can give me some hints. Is it reasonable to start with building pypy > with psycopg support from here > https://bitbucket.org/alex_gaynor/pypy-postgresql and then try to > merge from upstream? > > 2012/5/27 Gelonida N : > > Hi, > > > > Is there any good document / tutorial showing how to build pypy with the > > Rpython version of psycopg? > > > > Thanks an advance for good links. > > > > I'm working on Ubuntu 12.04 > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Hi There is a branch called psycopg2-ctypes link: https://github.com/mvantellingen/psycopg2-ctypes Because pypy supports ctypes well so I guess this branch should work with pypy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Mon May 28 21:00:31 2012 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 28 May 2012 22:00:31 +0300 Subject: [pypy-dev] test_rpoll.py::test_translate with MinGW32 In-Reply-To: References: Message-ID: <4FC3CB4F.1060507@gmail.com> On 28/05/2012 6:04 PM, bookaa wrote: > with > set cc=mingw32-gcc > this test fail: > pytest.py pypy/rlib/test/test_rpoll.py::test_translate > this is how to patch: > > diff -crN pypy-pypy-4a38b43757e3/pypy/rlib/_rsocket_rffi.py > pypy-pypy-4a38b43757e3.bookaa/pypyhttp://mail.python.org/mailman/listinfo/pypy-dev Thanks for the mingw32 work. I have committed this patch to the default pypy branch. As an alternative to posting patches here, please consider using bugs.pypy.org or a bitbucket branch or fork. Also, using 'hg diff' for patch formats would greatly simplify testing and committing them. If you need help getting started please join us on IRC at #pypy. Matti From matti.picus at gmail.com Mon May 28 23:40:53 2012 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 29 May 2012 00:40:53 +0300 Subject: [pypy-dev] test__ffi_call_releases_gil test failing in win32 Message-ID: <4FC3F0E5.5010201@gmail.com> I need some hints how to debug a crash in pypy when running this test test__ffi_call_releases_gil,in pypy.module.pypyjit.test_pypy_c.test__ffi. I fixed the failing test so it can find the Sleep function (in win32) but then pypy crashes, with one of those annoying "report this error" dialog boxes. Here's what I have so far:https://gist.github.com/2821258 It's python code that is run with pypy --jit threshold=150 When run without the --jit, all is OK But with the jit flag pypy (win32, of course :) ) crashes Here is the PYPYLOG: https://gist.github.com/2821307 From amauryfa at gmail.com Tue May 29 00:05:44 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 29 May 2012 00:05:44 +0200 Subject: [pypy-dev] test__ffi_call_releases_gil test failing in win32 In-Reply-To: <4FC3F0E5.5010201@gmail.com> References: <4FC3F0E5.5010201@gmail.com> Message-ID: 2012/5/28 Matti Picus > I need some hints how to debug a crash in pypy when running this test > test__ffi_call_releases_gil,in pypy.module.pypyjit.test_pypy_** > c.test__ffi. > I fixed the failing test so it can find the Sleep function (in win32) but > then pypy crashes, with one of those annoying "report this error" dialog > boxes. > Here's what I have so far:https://gist.github.com/**2821258 > It's python code that is run with pypy --jit threshold=150 > When run without the --jit, all is OK > But with the jit flag pypy (win32, of course :) ) crashes > Here is the PYPYLOG: https://gist.github.com/**2821307 A wild guess: Sleep is a WINAPI function, with a calling convention different than the default C convention. Maybe this makes a difference with the ffi module? -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Tue May 29 10:15:36 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 29 May 2012 10:15:36 +0200 Subject: [pypy-dev] test__ffi_call_releases_gil test failing in win32 In-Reply-To: References: <4FC3F0E5.5010201@gmail.com> Message-ID: <4FC485A8.8080108@gmail.com> On 05/29/2012 12:05 AM, Amaury Forgeot d'Arc wrote: > A wild guess: Sleep is a WINAPI function, with a calling convention different than > the default C convention. Maybe this makes a difference with the ffi module? I bet that the problem is inside the implementation of CALL_RELEASE_GIL in the x86 backend. It used to fail on windows for exactly this reason (wrong calling convention), but IIRC Armin fixed it during the last gothenburg sprint, although I can't find the relevant checkin. Armin, do you remember? ciao, Anto From arigo at tunes.org Tue May 29 10:28:00 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 29 May 2012 10:28:00 +0200 Subject: [pypy-dev] test__ffi_call_releases_gil test failing in win32 In-Reply-To: <4FC485A8.8080108@gmail.com> References: <4FC3F0E5.5010201@gmail.com> <4FC485A8.8080108@gmail.com> Message-ID: Hi, On Tue, May 29, 2012 at 10:15 AM, Antonio Cuni wrote: >> A wild guess: Sleep is a WINAPI function, with a calling convention different than >> the default C convention. Maybe this makes a difference with the ffi module? Yes, exactly. I don't see anything related to that in module/_ffi too, which means it's broken in this respect (or just doesn't support WINAPI functions, which is probably not intended). > but IIRC Armin fixed it during the last gothenburg sprint, > although I can't find the relevant checkin. See how _rawffi/interp_rawffi.py takes a 'flags' argument from app-level and passes it to clibffi in the call to RawFuncPtr(). A bient?t, Armin. From santagada at gmail.com Tue May 29 16:38:54 2012 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 29 May 2012 11:38:54 -0300 Subject: [pypy-dev] Python FFI In-Reply-To: References: <201205230429.q4N4TebX003789@theraft.openend.se> Message-ID: I think that what laura meant was that pypy people should talk to luajit people, because if someone came to talk to pypy about ctypes (the other big ffi for a dynamic language) pypy devs would have said "don't copy us because in pratictice this design is bad.". As you said that you talked to luajit author this subject is settled. On Sun, May 27, 2012 at 12:54 PM, Armin Rigo wrote: >> Sounds great to me. ?Except that Mike Pell (LuaJit's author) hasn't >> been cc'd ?on this note. > > He has shown up in #pypy to discuss it. > >>?Because, dear God, >> if he had not already existed and made this thing, then *we* >> would be the people with the best working FFI. > > I want to emphasis here that it's done "the LuaJIT way". ?I don't want > to pretend having invented anything myself. ?If it were not for > LuaJIT, then very probably we would not have found this solution --- > although it seems very natural in hindsight. > >> No NO NO -- our way was an experiment and too messy to be used. > > Unsure what you mean here... ?I've used the Python ffi interface on > some examples, and it seems both clean enough and much more natural > than packaging tons of objects that you have to learn about first. ?It > is maybe as close to a no-API interface that you can get, provided you > already know Python and have a basic knowledge of C. > > Example: http://bpaste.net/show/30171/ ? This is as close to the > original C code as possible, with a small page of C declarations > copy-pasted from the man pages. ?I think that this example alone shows > that the approach really works, although it should be tried on larger > examples too. -- Leonardo Santagada From robert.zaremba at zoho.com Tue May 29 15:49:37 2012 From: robert.zaremba at zoho.com (Robert Zaremba) Date: Tue, 29 May 2012 13:49:37 +0000 (UTC) Subject: [pypy-dev] Python FFI References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Whats the problem with cppyy? Why don't extend it rather than build new solution based on pycparser? From anto.cuni at gmail.com Wed May 30 10:07:46 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 30 May 2012 10:07:46 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? Message-ID: <4FC5D552.9050002@gmail.com> Hi all, after some months of work on the py3k branch, I realized that the current strategy/workflow does not scale well and thus I'd like to change it. For those who are not aware, currently we have the default branch where the main development is done, and which includes code for both the rpython translator toolchain and the python 2 interpreter. The py3k branch does not touch the translator toolchain, but modifies the python 2 interpreter to make it a py3k interpreter. These changes are actually destroying the python 2 semantics, which means that the long term goal is to never merge py3k to default, but keeping the development in parallel, and regularly merge default into py3k to make sure that py3k gets benefits of the various improvements in the JIT, GC, etc. In the past months, I ended up spending a considerable amount of time in resolving merge conflicts. This happens all the time that someone modifies something in the python 2 interpreter, for example to apply a new cool optimizations. While on the one hand it is cool to automatically have new cool optimizations on py3k, on the other hand it is blocker which stops me to work on it effectively. After a bit of discussion on IRC, I propose to solve this problem by detaching the development of the py3k interpreter from the development of the python 2 one. Pros: - faster development of py3k - lower entry barrier for new contributors, because the relationship between the various parts will be much simpler - it will be straightforward to apply the new features of the translator toolchain to the py3k branch - it will be easier to split the toolchain from the actual interpreter the day we will finally decide to do it Cons: - we will need to manually port the optimizations done in the interpreter on the default branch to py3k. Note that right now it's now really "automatic" anyway, because merging is painful. If we decide to go for this route, the next question is: where to store the code? I think there are two main solutions: 1) add a new "pypy/py3k" directory where to copy all the relevant modules. E.g. pypy/py3k/interpreter, pypy/py3k/objspace/std, "pypy/py3k/modules. 2) start a completely new repository which contains only the code for py3k. Solution (2) is better and cleaner in theory. However I fear it would soon become a mess to handle, because every change in the translator toolchain would potentially break py3k. I don't want a situation in which we say "yes, you can build py3k but only if you take revision XXX and you use revision YYY of the toolchain, unless the phase of the moon is empty". Solution (1) is more practical and it would probably lead to less problems in the short term. For now, I would still keep the code in the py3k branch, so the normal development of pypy would not be affected. Before doing it, I'd like to hear opinions and comments, in particular of people who already worked on py3k and/or are generally interested in it. Please be constructive :-). ciao, Anto From amauryfa at gmail.com Wed May 30 11:23:02 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 30 May 2012 11:23:02 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: <4FC5D552.9050002@gmail.com> References: <4FC5D552.9050002@gmail.com> Message-ID: 2012/5/30 Antonio Cuni > 2) start a completely new repository which contains only the code for py3k. How is this different from the current py3k branch? We could also just decide to never merge the default branch, or merge only after a release of the main PyPy version. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Wed May 30 11:42:31 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 30 May 2012 11:42:31 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: References: <4FC5D552.9050002@gmail.com> Message-ID: <4FC5EB87.3030803@gmail.com> Hi Amaury, On 05/30/2012 11:23 AM, Amaury Forgeot d'Arc wrote: > 2012/5/30 Antonio Cuni > > > 2) start a completely new repository which contains only the code for py3k. > > > How is this different from the current py3k branch? the difference is that you would get the improvements in translator toolchain for free. See also my point below. > We could also just decide to never merge the default branch, > or merge only after a release of the main PyPy version. possibly, but delaying the merge would make it even more painful. The risk is that it'll become so painful that nobody will feel like doing it, and thus we diverge more and more. At the end, we end up with a py3k branch which can't make use of the cool new features of the JIT/GC/etc. and that will always lack behind python 2. Another point of view is that IMHO porting the changes by doing merges is harder/more time consuming than porting them by hand. ciao, Anto From fijall at gmail.com Wed May 30 11:49:34 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 30 May 2012 11:49:34 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: <4FC5EB87.3030803@gmail.com> References: <4FC5D552.9050002@gmail.com> <4FC5EB87.3030803@gmail.com> Message-ID: On Wed, May 30, 2012 at 11:42 AM, Antonio Cuni wrote: > Hi Amaury, > > On 05/30/2012 11:23 AM, Amaury Forgeot d'Arc wrote: > > 2012/5/30 Antonio Cuni >> > > > > 2) start a completely new repository which contains only the code > for py3k. > > > > > > How is this different from the current py3k branch? > > the difference is that you would get the improvements in translator > toolchain > for free. See also my point below. > > > We could also just decide to never merge the default branch, > > or merge only after a release of the main PyPy version. > > possibly, but delaying the merge would make it even more painful. The risk > is > that it'll become so painful that nobody will feel like doing it, and thus > we > diverge more and more. At the end, we end up with a py3k branch which can't > make use of the cool new features of the JIT/GC/etc. and that will always > lack > behind python 2. > > Another point of view is that IMHO porting the changes by doing merges is > harder/more time consuming than porting them by hand. > > ciao, > Anto > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Hi Anto. I think 1) is a no-no for me. This would first mean we have py3k in the default checkout (why???) and also that we need to make sure that py3k tests pass all the time (they don't pass to start with). I don't see this being any beneficial to the current model. Besidies, this also means we'll never upgrade rpython to py3k (which might be a good thing, just saying). Overall I'm very against pushing *any* burden towards other pypy devs, we have quite enough work. How about you start with detaching interpreter and translation toolchain so those things can leave separately? Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Wed May 30 12:08:39 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 30 May 2012 12:08:39 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: References: <4FC5D552.9050002@gmail.com> <4FC5EB87.3030803@gmail.com> Message-ID: <4FC5F1A7.7040405@gmail.com> On 05/30/2012 11:49 AM, Maciej Fijalkowski wrote: > Hi Anto. > > I think 1) is a no-no for me. This would first mean we have py3k in the > default checkout (why???) and also that we need to make sure that py3k tests > pass all the time (they don't pass to start with). I don't see this being any > beneficial to the current model. no, it would leave in the py3k branch. No merging to default unless there is a consensus. > Besidies, this also means we'll never upgrade rpython to py3k (which might be > a good thing, just saying). nothing would stops us to take the interpreter/ from py3k and port rpython to python 3, although I'm not sure if it would be a good idea (euphemism :-)). But this is orthogonal to this discussion. > Overall I'm very against pushing *any* burden > towards other pypy devs, we have quite enough work. keeping it in the py3k branch would not change anything for people who don't care about py3k. That's why I asked for opinions of people who care :-) > How about you start with > detaching interpreter and translation toolchain so those things can leave > separately? no. This would be too much work for little benefit from the py3k point of view. Of course if it happens independently then py3k would benefit of it, but the task itself is not on top of my priorities. ciao, Anto From amauryfa at gmail.com Wed May 30 12:53:22 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 30 May 2012 12:53:22 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: <4FC5EB87.3030803@gmail.com> References: <4FC5D552.9050002@gmail.com> <4FC5EB87.3030803@gmail.com> Message-ID: 2012/5/30 Antonio Cuni > Another point of view is that IMHO porting the changes by doing merges is > harder/more time consuming than porting them by hand. > We probably don't have the same view on merges then :) I consider that when a merge is successful (no conflict), it's a win. And conflicts are markers to say "hey, port this change by hand". Of course, it would be better if we could just merge the translator/ or jit/ directories on a regular basis, and come back later to merge (or port) changes from the interpreter/ and objspace/ directories. But hg does not seem to allow this. I estimate that I spent ~2h on each merge from default to py3k. If one merge per month is enough, it's a task I can definitely find time for. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From gelonida at gmail.com Wed May 30 13:17:46 2012 From: gelonida at gmail.com (Gelonida N) Date: Wed, 30 May 2012 13:17:46 +0200 Subject: [pypy-dev] first test with pypy and django Message-ID: Hi, I wanted to make my very first test with pypy by using and django and sqlite3 when running ./manage.py syncdb I'm asked to enter a password and pypy aborts as soon as I press enter. > File "/home/klausf/mypypy/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 109, in handle > password = getpass.getpass() > File "/usr/lib/pypy/lib-python/2.7/getpass.py", line 74, in unix_getpass > stream.flush() # issue7208 > IOError: [Errno 29] Illegal seek: '' Version info: Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08) [PyPy 1.8.0 with GCC 4.6.2] Django version: 1.3.1 I even found a bug, which seems related ( https://bugs.pypy.org/issue872 ) However I don't understand the impact and what this means exactly. Should I try to install another pypy (and if yes, how can I install it next to an existing one on Ubuntu 12.04)? Thanks in advance for your help. From anto.cuni at gmail.com Wed May 30 13:35:48 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 30 May 2012 13:35:48 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: References: <4FC5D552.9050002@gmail.com> <4FC5EB87.3030803@gmail.com> Message-ID: <4FC60614.4030505@gmail.com> On 05/30/2012 12:53 PM, Amaury Forgeot d'Arc wrote: > We probably don't have the same view on merges then :) > I consider that when a merge is successful (no conflict), it's a win. > And conflicts are markers to say "hey, port this change by hand". yes, I also consider it a win. However, it happens rarely. Maybe it's just me that can't handle them, but conflicts markers are usually put in places which makes it very difficult to understand what's going on. I always end up at looking at the unchanged files before the merge and the diff in the branch, forgetting the markers. > Of course, it would be better if we could just merge the translator/ or jit/ > directories on a regular basis, > and come back later to merge (or port) changes from the interpreter/ and > objspace/ directories. > But hg does not seem to allow this. yes. That would be perfect, but we can't :-( > I estimate that I spent ~2h on each merge from default to py3k. > If one merge per month is enough, it's a task I can definitely find time for. My impression is that the time spent for merges is increasing, and this is not surprising because the two branches are slowly diverging. Anyway, if you volunteer to do the merges regularly I won't certainly stop you :-). We can always do the split later. ciao, Anto From arigo at tunes.org Wed May 30 15:30:47 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 May 2012 15:30:47 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi Robert, On Tue, May 29, 2012 at 3:49 PM, Robert Zaremba wrote: > Whats the problem with cppyy? > Why don't extend it rather than build new solution based on pycparser? Once again the design goals are different. Note that the first mail in this thread was 15 days ago. During this time we got a working prototype. I'm sure that it would have taken us 15 days too to even figure out how to correctly interface with cppyy, not to mention that it requires a C++ compiler and --- as far as I know --- cannot handle some of the most C-specific ways headers are written, like (ab)uses of macros. A bient?t, Armin. From arigo at tunes.org Wed May 30 16:04:52 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 May 2012 16:04:52 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: <4FC60614.4030505@gmail.com> References: <4FC5D552.9050002@gmail.com> <4FC5EB87.3030803@gmail.com> <4FC60614.4030505@gmail.com> Message-ID: Hi Antonio, On Wed, May 30, 2012 at 1:35 PM, Antonio Cuni wrote: >> Of course, it would be better if we could just merge the translator/ or jit/ >> directories on a regular basis, >> and come back later to merge (or port) changes from the interpreter/ and >> objspace/ directories. >> But hg does not seem to allow this. > > yes. That would be perfect, but we can't :-( Then we can probably arrange things so that we use "translate.py" from default, and not from the "py3k branch", which would be stripped of the translation parts. More precisely, we could organize this "py3k branch" --- quotes, because likely living then in another repo --- with an only marginally different directory structure: e.g. call the top-level directory "py3k" instead of "pypy". Then you would use the default's "translate.py" to translate it, without getting conflicts between "pypy.interpreter" as used by translate.py and the new "py3k.interpreter" containing what you are translating. Of course the directories that would be in the py3k package would still have the same name as their original ones, so that we keep open the possibility to do merges without adding yet another layer of troubles. A bient?t, Armin. From wlavrijsen at lbl.gov Wed May 30 16:57:52 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 30 May 2012 07:57:52 -0700 (PDT) Subject: [pypy-dev] Python FFI In-Reply-To: References: <20120515192251.13157.1684105672.divmod.xquotient.168@localhost6.localdomain6> Message-ID: Hi, On Wed, 30 May 2012, Armin Rigo wrote: > On Tue, May 29, 2012 at 3:49 PM, Robert Zaremba wrote: >> Whats the problem with cppyy? >> Why don't extend it rather than build new solution based on pycparser? > Once again the design goals are different. right; to Robert: think (conceptually) of cppyy as "c++types". It has so far been presented in conjunction with Reflex, because the only way that it is useful for a Python programmer is with an automatic backend to fill in all necessary details. But for PyPy development purposes, the dispatch (cppyy) should be seen separately from the specification (Reflex/CINT/Cling/...) of C++ details. > Note that the first mail > in this thread was 15 days ago. During this time we got a working > prototype. I'm sure that it would have taken us 15 days too to even > figure out how to correctly interface with cppyy Actually, with Reflex it would be a bit of a hassle (you'd have to stuff the string in a file, add the required headers, run gccxml, etc., etc.) but with CINT or Cling, you could just hand the string to the C++ interpreter, run it and have the bindings generated on the fly. > not to mention that it requires a C++ compiler Yes, you simply can't develop something this clean and dependency free for C++. Of course, it'd work for a subset of C++, but that'd have limited use. > and --- as far as I know --- cannot handle > some of the most C-specific ways headers are written, like (ab)uses of > macros. Oh yes it can! :) If the compiler can parse it, then bring it on. :) Now I feel like providing bindings to cpp macro's just to prove that it can be done (did so in a limited way for PyROOT; haven't bothered for cppyy yet). Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From lac at openend.se Wed May 30 18:07:23 2012 From: lac at openend.se (Laura Creighton) Date: Wed, 30 May 2012 18:07:23 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: Message from Armin Rigo of "Wed, 30 May 2012 16:04:52 +0200." References: <4FC5D552.9050002@gmail.com> <4FC5EB87.3030803@gmail.com> <4FC60614.4030505@gmail.com> Message-ID: <201205301607.q4UG7NMD006383@theraft.openend.se> About 25 years ago (dear god that makes me feel old) when CVS was the new and happening thing in version control systems, we were in the really bad problem area at Human Computing Resources of porting unix to more than 1000, kid you not, of different M68010, M68020, NS16032, NS32032, Zilog-I-forget, MIPS, and I forget what other hardware-based systems, with completely crazy combination of graphical cards, interrupt driven weird hardware and well, trust me, it was a mess. We wanted one kernel that we could deploy everywhere. This was impossible, but we got the best we could do, given our constraints by changing the whole build system to one in which you typed 'make' and it started by copying all the files from our home repository system to a complete and brand new file system, and then copying all the tests to the same system, and then running the tests that ran before you tried to compile a kernel, (mostly having to do with seeing that your hardware was sane and that the setup script worked, and was happy with the results) and then you made your kernel. Which could take more than an hour. And then you ran your kernel tests against the new kernel (which happened overnight). In the morning we had a whole new set of errors to deal with. I don't think that we have reached this level of complexity with PyPy yet, but it is something to think about. The time of 'everything is a i386' is over now, and it is not clear how this will play out in the realm of systems and even application programming languages. Clearly not as badly as we had it when we were making kernels as quickly as we were being sent hardware, but there still is some effect. Laura From robert.zaremba at zoho.com Wed May 30 18:00:16 2012 From: robert.zaremba at zoho.com (Robert Zaremba) Date: Wed, 30 May 2012 18:00:16 +0200 Subject: [pypy-dev] Python FFI In-Reply-To: References: Message-ID: <2392225.RKfMf6lBSA@robert-zaremba-bc> Thanks for clarification. I've been wondering why to dobule the work. I'm also really fascinated about this idea, and that i don't need to use separate tool. It looks really great! Thanks! -- Robert From faassen at startifact.com Wed May 30 19:24:24 2012 From: faassen at startifact.com (Martijn Faassen) Date: Wed, 30 May 2012 19:24:24 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: <4FC5D552.9050002@gmail.com> References: <4FC5D552.9050002@gmail.com> Message-ID: Hi there, Just throwing in my little bit: any change that is made that would make it easier to run Python 2 and Python 3 interpretors in the same process would interesting, as I'm still vaguely dreaming (nothing more) of a combined interpreter that can run both Python 2 and Python 3 code. Regards, Martijn From fijall at gmail.com Wed May 30 20:42:30 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 30 May 2012 20:42:30 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: References: <4FC5D552.9050002@gmail.com> Message-ID: On Wed, May 30, 2012 at 7:24 PM, Martijn Faassen wrote: > Hi there, > > Just throwing in my little bit: any change that is made that would > make it easier to run Python 2 and Python 3 interpretors in the same > process would interesting, as I'm still vaguely dreaming (nothing > more) of a combined interpreter that can run both Python 2 and Python > 3 code. > > Regards, > > Martijn Hi Martijn. Can you describe what sort of semantics you have in mind? Would you like to have two copies of builtin modules? How about namespaces? What about objects being passed from one interpreter to the another? Would they magically change or would they be "py2k dict" and "py3k dict"? If you can describe the semantics of a proposed beast I'm willing to answer how likely it is to happen Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: From gelonida at gmail.com Wed May 30 23:42:46 2012 From: gelonida at gmail.com (Gelonida N) Date: Wed, 30 May 2012 23:42:46 +0200 Subject: [pypy-dev] IOError with getpass() in django Message-ID: <4FC69456.7080408@gmail.com> Hi, I reposted as I noticed, that my inital subject line doesn't really indicate what's the issue. ------------------------------------------------------------------- I wanted to make my very first test with pypy by using django and sqlite3 on a Linux host (Ubuntu 12.04) when running ./manage.py syncdb I'm asked to enter a password and pypy aborts as soon as I press enter. > File "/home/klausf/mypypy/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 109, in handle > password = getpass.getpass() > File "/usr/lib/pypy/lib-python/2.7/getpass.py", line 74, in unix_getpass > stream.flush() # issue7208 > IOError: [Errno 29] Illegal seek: '' Version info: Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08) [PyPy 1.8.0 with GCC 4.6.2] Django version: 1.3.1 I even found a bug, which seems related ( https://bugs.pypy.org/issue872 ) However I don't understand the impact and what this means exactly. Should I try to install another pypy (and if yes, how can I install it next to an existing one on Ubuntu 12.04)? Thanks in advance for your help. From gelonida at gmail.com Wed May 30 23:45:16 2012 From: gelonida at gmail.com (Gelonida N) Date: Wed, 30 May 2012 23:45:16 +0200 Subject: [pypy-dev] how to build pypy with psycopg (links / requirements) In-Reply-To: References: Message-ID: On 05/28/2012 05:09 PM, gelin yan wrote: > > > On Mon, May 28, 2012 at 11:03 PM, ????? ??????? > > wrote: > > If there is no such document, I am willing to try to make psycopg work > with pypy, if it is a suitable work for pypy newbie, and if someone > can give me some hints. Is it reasonable to start with building pypy > with psycopg support from here > https://bitbucket.org/alex_gaynor/pypy-postgresql and then try to > merge from upstream? > > 2012/5/27 Gelonida N >: > > Hi, > > > > Is there any good document / tutorial showing how to build pypy > with the > > Rpython version of psycopg? . . . > > > Hi > > There is a branch called psycopg2-ctypes > > link: https://github.com/mvantellingen/psycopg2-ctypes > > Because pypy supports ctypes well so I guess this branch should work > with pypy. > > Yeah I can try this. I'm just afraid, that this library will not be efficient. The main intention of using pypy would be to speed up my application. It might still be, that the overall application will be sped up, but it's a pity gaining in template rendering and loosing in SQL accesses. I will try this as soon as I get basic django up and running on my host (see my other post, 'problems with getpass()' From Ronny.Pfannschmidt at gmx.de Thu May 31 00:24:11 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 31 May 2012 00:24:11 +0200 Subject: [pypy-dev] how to build pypy with psycopg (links / requirements) In-Reply-To: References: Message-ID: <4FC69E0B.5060706@gmx.de> On 05/30/2012 11:45 PM, Gelonida N wrote: > On 05/28/2012 05:09 PM, gelin yan wrote: >> >> >> On Mon, May 28, 2012 at 11:03 PM, ????? ??????? >> > wrote: >> >> If there is no such document, I am willing to try to make psycopg work >> with pypy, if it is a suitable work for pypy newbie, and if someone >> can give me some hints. Is it reasonable to start with building pypy >> with psycopg support from here >> https://bitbucket.org/alex_gaynor/pypy-postgresql and then try to >> merge from upstream? >> >> 2012/5/27 Gelonida N >: >> > Hi, >> > >> > Is there any good document / tutorial showing how to build pypy >> with the >> > Rpython version of psycopg? > > . . . >> >> >> Hi >> >> There is a branch called psycopg2-ctypes >> >> link: https://github.com/mvantellingen/psycopg2-ctypes >> >> Because pypy supports ctypes well so I guess this branch should work >> with pypy. >> >> > Yeah I can try this. I'm just afraid, that this library will not be > efficient. > The main intention of using pypy would be to speed up my application. > It might still be, that the overall application will be sped up, but > it's a pity gaining in template rendering and loosing in SQL accesses. note that cpython api base extension are slow due to the cpython refcounting/api emulation it seems quite possible that a ctypes version will get faster someone should measure > > I will try this as soon as I get basic django up and running on my host > (see my other post, 'problems with getpass()' > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From alex.pyattaev at gmail.com Thu May 31 01:09:20 2012 From: alex.pyattaev at gmail.com (Alex Pyattaev) Date: Thu, 31 May 2012 02:09:20 +0300 Subject: [pypy-dev] cppyy use case question Message-ID: <21606049.Qf17BoIqhp@hunter-laptop> Hello, I am considering trying cppyy on a large project, and I have a question. I need to be able to add a reference to Python objects into the c++ classes. In essence: #Python code: x="My python object" inst = cpp_class() inst.add_data(x) inst.do_stuff()#This does not modify x in any way z=inst.get_data(x) So, cpp_class acts as a container for python objects, and it does not need to know what kind of object is inside. I was able to make it work with pypy through basic C API, using SWIG and typemaps by mapping void* into PyObject*, but that is somewhat hackish. I am now thinking how would the same logic look with cppyy extension. PS: building cppyy is pain in the ass... You guys seriously need to think about build logs =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ronny.Pfannschmidt at gmx.de Thu May 31 01:14:09 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 31 May 2012 01:14:09 +0200 Subject: [pypy-dev] on using the new bitbucket teams feature Message-ID: <4FC6A9C1.5040806@gmx.de> hi, since bitbucket finally supports teams, i think should look into migrating the pypy user to that feature -- Ronny From wlavrijsen at lbl.gov Thu May 31 03:52:25 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 30 May 2012 18:52:25 -0700 (PDT) Subject: [pypy-dev] cppyy use case question In-Reply-To: <21606049.Qf17BoIqhp@hunter-laptop> References: <21606049.Qf17BoIqhp@hunter-laptop> Message-ID: Hi Alex, > So, cpp_class acts as a container for python objects, and it does not need to > know what kind of object is inside. I was able to make it work with pypy > through basic C API, using SWIG and typemaps by mapping void* into PyObject*, > but that is somewhat hackish. I am now thinking how would the same logic look > with cppyy extension. this should be supported directly (PyROOT does), but where in CPython handing a PyObject* is trivial, in PyPy one has to be created. My best guess is that the route would be cpyext to get such a PyObject*, then hand that over. I don't think it can be hacked around (a void* argument will peel the contained C++ object, which will fail, since it's a real Python object). Identity preservation (on the return) should be easy, and the C++ side should keep the refcount up for the proper duration. Let me figure that one out. Is as good a reason as any to have some fun and do some coding again for the first time in weeks. :P > PS: > building cppyy is pain in the ass... You guys seriously need to think about > build logs =) Sorry (but this is why at CERN we have a global file system (afs) and simply install there for all users :) ) ... So, is there anything in the instructions that was missing and needs to be added? Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From kostia.lopuhin at gmail.com Thu May 31 07:18:45 2012 From: kostia.lopuhin at gmail.com (=?KOI8-R?B?68/T1NEg7M/Q1cjJzg==?=) Date: Thu, 31 May 2012 09:18:45 +0400 Subject: [pypy-dev] how to build pypy with psycopg (links / requirements) In-Reply-To: <4FC69E0B.5060706@gmx.de> References: <4FC69E0B.5060706@gmx.de> Message-ID: In my app I observe that after jit warm up the speed is on par with psycopg2 in CPython, and about 1.5 times slower before warm-up - but that is just one case, may be different with different patterns. And I tested only reads. 2012/5/31 Ronny Pfannschmidt : > > note that cpython api base extension are slow due to the cpython > refcounting/api emulation > it seems quite possible that a ctypes version will get faster > > someone should measure > > From mail at justinbogner.com Thu May 31 07:20:08 2012 From: mail at justinbogner.com (Justin Bogner) Date: Wed, 30 May 2012 23:20:08 -0600 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: <4FC5D552.9050002@gmail.com> (Antonio Cuni's message of "Wed, 30 May 2012 10:07:46 +0200") References: <4FC5D552.9050002@gmail.com> Message-ID: <877gvtq647.fsf@glimpse.justinbogner.com> Antonio Cuni writes: > after some months of work on the py3k branch, I realized that the current > strategy/workflow does not scale well and thus I'd like to change it. > > For those who are not aware, currently we have the default branch where the > main development is done, and which includes code for both the rpython > translator toolchain and the python 2 interpreter. > > The py3k branch does not touch the translator toolchain, but modifies the > python 2 interpreter to make it a py3k interpreter. > If we decide to go for this route, the next question is: where to store the > code? I think there are two main solutions: > > 1) add a new "pypy/py3k" directory where to copy all the relevant modules. > E.g. pypy/py3k/interpreter, pypy/py3k/objspace/std, "pypy/py3k/modules. > > 2) start a completely new repository which contains only the code for py3k. Any solution where improvements to the translator have to be ported manually is a recipe for trouble. The translator toolchain is and should always be shared. The interpreters, on the other hand, are where the merge difficulties come from. These are fundamentally diverging since the python 2 and 3 languages are actually quite different. Because of these two things, there are only two sane solutions, and (2) above is not one of them. Adding a py3k interpreter to the default branch is reasonable, and could lead to sharing common parts in a top level directory at some point, but it comes with a burden on the current python 2 development work. The other solution is to split the current pypy tree in two. Having a translator and an interpreter as separate repositories makes the translator more accessible as a tool, and projects to implement other languages' interpreters need only depend on it. This is the prettiest solution architecturally, but adds the burden on developers to match translator with compatible interpreters, which may or may not end up being a pain point. Anyways, a completely separate pypy3k repository is no better than the current situation, and I would argue that it's far worse. The pains you're experiencing trying to do merges today aren't even close to the kind of pain you'll experience trying to merge in interpreter improvements manually. From william.leslie.ttg at gmail.com Thu May 31 07:27:37 2012 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Thu, 31 May 2012 15:27:37 +1000 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: <877gvtq647.fsf@glimpse.justinbogner.com> References: <4FC5D552.9050002@gmail.com> <877gvtq647.fsf@glimpse.justinbogner.com> Message-ID: On 31 May 2012 15:20, Justin Bogner wrote: > The other solution is to split the current pypy tree in two. Having a > translator and an interpreter as separate repositories makes the > translator more accessible as a tool, and projects to implement other > languages' interpreters need only depend on it. This is the prettiest > solution architecturally, but adds the burden on developers to match > translator with compatible interpreters, which may or may not end up > being a pain point. Do remember that the translator actually requires the python 2 interpreter, that is a fundamental part of the way it works. So moving the translator into a different repository now also means maintaining two python 2 interpreters. -- William Leslie From mail at justinbogner.com Thu May 31 07:35:43 2012 From: mail at justinbogner.com (Justin Bogner) Date: Wed, 30 May 2012 23:35:43 -0600 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: (William ML Leslie's message of "Thu, 31 May 2012 15:27:37 +1000") References: <4FC5D552.9050002@gmail.com> <877gvtq647.fsf@glimpse.justinbogner.com> Message-ID: <87396grjyo.fsf@glimpse.justinbogner.com> William ML Leslie writes: > Do remember that the translator actually requires the python 2 > interpreter, that is a fundamental part of the way it works. So > moving the translator into a different repository now also means > maintaining two python 2 interpreters. Ah yes, bootstrapping makes everything difficult, doesn't it? From techtonik at gmail.com Thu May 31 07:52:15 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 31 May 2012 05:52:15 +0000 (UTC) Subject: [pypy-dev] AOSA book chapter References: Message-ID: Benjamin Peterson python.org> writes: > > Hello pypy-dev, > As some of you may know, this past year I wrote a chapter about PyPy > for Architecture of Open Source Applications Volume 2. It has brought > to my attention that my communication with the PyPy community about it > was poor. Since it deeply involved the PyPy project, I should have > been more public in its writing; I should have sent it to pypy-dev for > review and comments. Now that this has been pointed out to me, it's > painfully obvious that I fell short. > > Next time such an opportunity arises, I look forward to working fully > with the PyPy community on it. > > Please accept my apologies, > Benjamin > Hi Benjamin, That's nice to read the description of PyPy innards. Great work! Is seems possible to reuse this text and enhance it for PyPy docs, isn't it? So the opportunity is not lost. =) In the end it could become as visually entertaining as NaCl stuff: https://developers.google.com/native-client/dev/overview Looking forward to more pictures. =) From fijall at gmail.com Thu May 31 09:15:54 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 31 May 2012 09:15:54 +0200 Subject: [pypy-dev] IOError with getpass() in django In-Reply-To: <4FC69456.7080408@gmail.com> References: <4FC69456.7080408@gmail.com> Message-ID: On Wed, May 30, 2012 at 11:42 PM, Gelonida N wrote: > Hi, > > I reposted as I noticed, that my inital subject line doesn't really > indicate what's the issue. > ------------------------------**------------------------------**------- > > I wanted to make my very first test with pypy by using django and sqlite3 > on a Linux host (Ubuntu 12.04) > > when running ./manage.py syncdb I'm asked to enter a password and pypy > aborts as soon as I press enter. > > File "/home/klausf/mypypy/site-**packages/django/contrib/auth/** >> management/commands/**createsuperuser.py", line 109, in handle >> password = getpass.getpass() >> File "/usr/lib/pypy/lib-python/2.7/**getpass.py", line 74, in >> unix_getpass >> stream.flush() # issue7208 >> IOError: [Errno 29] Illegal seek: '' >> > > > Version info: > Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08) > [PyPy 1.8.0 with GCC 4.6.2] > > Django version: 1.3.1 > > > I even found a bug, which seems related ( https://bugs.pypy.org/issue872 ) > > However I don't understand the impact and what this means exactly. > Should I try to install another pypy (and if yes, how can I install it > next to an existing one on Ubuntu 12.04)? > > Thanks in advance for your help. > No, it's a bug and we did not fix it yet. I'll have a look some time today. -------------- next part -------------- An HTML attachment was scrubbed... URL: From william.leslie.ttg at gmail.com Thu May 31 10:06:18 2012 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Thu, 31 May 2012 18:06:18 +1000 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: References: <4FC5D552.9050002@gmail.com> Message-ID: On 31 May 2012 04:42, Maciej Fijalkowski wrote: > On Wed, May 30, 2012 at 7:24 PM, Martijn Faassen > wrote: >> >> Hi there, >> >> Just throwing in my little bit: any change that is made that would >> make it easier to run Python 2 and Python 3 interpretors in the same >> process would interesting, as I'm still vaguely dreaming (nothing >> more) of a combined interpreter that can run both Python 2 and Python >> 3 code. >> >> Regards, >> >> Martijn > > > Hi Martijn. > > Can you describe what sort of semantics you have in mind? Would you like to > have two copies of builtin modules? How about namespaces? What about objects > being passed from one interpreter to the another? Would they magically > change or would they be "py2k dict" and "py3k dict"? If you can describe the > semantics of a proposed beast I'm willing to answer how likely it is to > happen I think we already discussed this at one point, here is what I remember getting out of it: * Any such language integration that we do encourages people to write pypy-only programs. There was a question as to whether this was a good idea. I think someone suggested it could go further than python 2/3 and allow interaction with scheme or prolog or javascript since we are already there. * There probably are arguments around semantics, but any solution is better than no solution. This is a good topic for further research imao. * It is worthwhile considering the effect it has on python 3 uptake and porting. If pypy gave people an easy way out, it could have made quite a mess. I don't think this is as significant a problem now as it has been. And of course if you don't want to do language integration, but just add eg a command line switch, you're not getting much out of it but the cost is significant, it means users have to co-ordinate the upgrades of two languages, it increases translation and testing time, etc. ---------------------------------------- By the way, I like solution 1; it's a bit closer to the way pypy/lang was done. I get the cases for moving the other languages away, but python 3 is different because so much of the existing code can be re-used. -- William Leslie From gelonida at gmail.com Thu May 31 10:15:45 2012 From: gelonida at gmail.com (Gelonida N) Date: Thu, 31 May 2012 10:15:45 +0200 Subject: [pypy-dev] IOError with getpass() in django In-Reply-To: References: <4FC69456.7080408@gmail.com> Message-ID: On 05/31/2012 09:15 AM, Maciej Fijalkowski wrote: > On Wed, May 30, 2012 at 11:42 PM, Gelonida N > wrote: > > I wanted to make my very first test with pypy by using django and > sqlite3 on a Linux host (Ubuntu 12.04) > > when running ./manage.py syncdb I'm asked to enter a password and > pypy aborts as soon as I press enter. > > File > "/home/gelonida/mypypy/site-__packages/django/contrib/auth/__management/commands/__createsuperuser.py", > line 109, in handle > password = getpass.getpass() > File "/usr/lib/pypy/lib-python/2.7/__getpass.py", line 74, in > unix_getpass > stream.flush() # issue7208 > IOError: [Errno 29] Illegal seek: '' > > Version info: > Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08) > [PyPy 1.8.0 with GCC 4.6.2] > > Django version: 1.3.1 > > > I even found a bug, which seems related ( > https://bugs.pypy.org/issue872 ) > > However I don't understand the impact and what this means exactly. > Should I try to install another pypy (and if yes, how can I install > it next to an existing one on Ubuntu 12.04)? > > Thanks in advance for your help. > > > No, it's a bug and we did not fix it yet. I'll have a look some time today. Thanks for your answer. Shouldn't this affect all users trying to use Django on a new project? I didn't find any way by means of command line arguments to avoid the call to getpass() How do others work around it? What I did in the end was, that I just called the first "./manage.py syncdb" with ordinary CPython. Subsequent calls won't call getpass() anymore and should thus be fine. From anto.cuni at gmail.com Thu May 31 10:29:52 2012 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 31 May 2012 10:29:52 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: References: <4FC5D552.9050002@gmail.com> <4FC5EB87.3030803@gmail.com> <4FC60614.4030505@gmail.com> Message-ID: <4FC72C00.4000508@gmail.com> On 05/30/2012 04:04 PM, Armin Rigo wrote: > Then we can probably arrange things so that we use "translate.py" from > default, and not from the "py3k branch", which would be stripped of > the translation parts. > > More precisely, we could organize this "py3k branch" --- quotes, > because likely living then in another repo --- with an only marginally > different directory structure: e.g. call the top-level directory > "py3k" instead of "pypy". Then you would use the default's > "translate.py" to translate it, without getting conflicts between > "pypy.interpreter" as used by translate.py and the new > "py3k.interpreter" containing what you are translating. > > Of course the directories that would be in the py3k package would > still have the same name as their original ones, so that we keep open > the possibility to do merges without adding yet another layer of > troubles. uhm, that's an interesting possibility, I didn't think of it. I wonder if mercurial handles merges well if we rename the top-level directory. To make things cleaner and easier to understand, we should probably also "hg rm" from py3k/ the directories which belongs to the toolchain, just to avoid confusion. I think that in this case at each merge mercurial would ask what to do with file X which has been deleted locally but changed remotely, but this is probably something that we can handle. As I said earlier, the drawback of such "decoupling" solutions is that as soon as you have two separate repos, you'll get troubles such as "you can translate revision XXX only if the pypy repo is at version YYY", which can be frustrating especially when you want to go back in the history. In theory mercurial subrepos are supposed to solve this problem, but in practice we should stay as far as we can from them :-(. Amaury: opinions on Armin's proposed solution? From alex.pyattaev at gmail.com Thu May 31 10:33:34 2012 From: alex.pyattaev at gmail.com (Alex Pyattaev) Date: Thu, 31 May 2012 11:33:34 +0300 Subject: [pypy-dev] cppyy use case question In-Reply-To: References: <21606049.Qf17BoIqhp@hunter-laptop> Message-ID: <1543846.T9Sm08LvAA@hunter-laptop> keskiviikko 30 toukokuu 2012 18:52:25 wlavrijsen at lbl.gov kirjoitti: > Hi Alex, > > > So, cpp_class acts as a container for python objects, and it does not need > > to know what kind of object is inside. I was able to make it work with > > pypy through basic C API, using SWIG and typemaps by mapping void* into > > PyObject*, but that is somewhat hackish. I am now thinking how would the > > same logic look with cppyy extension. > > this should be supported directly (PyROOT does), but where in CPython > handing a PyObject* is trivial, in PyPy one has to be created. My best > guess is that the route would be cpyext to get such a PyObject*, then hand > that over. > > I don't think it can be hacked around (a void* argument will peel the > contained C++ object, which will fail, since it's a real Python object). > > Identity preservation (on the return) should be easy, and the C++ side > should keep the refcount up for the proper duration. > > Let me figure that one out. Is as good a reason as any to have some fun and > do some coding again for the first time in weeks. :P > Thanks! I have accumulated some understanding about the process, and the issue can be worked around by having a dictionary that would map ints into python objects and vice versa before those are sent to c++, so when c++ functions feth them back they can be retreived. However, this would incur certain overhead, which may be rather annoying. Anyway, the whole point was to test if it will be any faster then cpyext, so unless this feature is properly resolved the test does not make any massive amount of sense (mostly due to this extra overhead). I'll probably run it anyway though=) Thanks for looking into it, its good to hear that somebody cares! > > PS: > > building cppyy is pain in the ass... You guys seriously need to think > > about > > build logs =) > > Sorry (but this is why at CERN we have a global file system (afs) and simply > install there for all users :) ) ... So, is there anything in the > instructions that was missing and needs to be added? > With respect to build, I installed root from gentoo repository, and the problem there is that headers are in /usr/include/root and the actual libs are in /usr/lib/root, but the pypy build expects libs to be in /usr/include/root/lib if you specify SYSROOT=/usr/include/root. So instead I ended up having to symlink those together to replicate whatever pypy build was expecting. This is indeed a hax, but probably if you install root from source package it is not needed. Anyway, it should be benefitial to add a header and library check in the beginning of the build to be certain that it does not fail after 20 mins of compiling. But for now symlink works just fine. With respect to genreflex command, you could probably warn people that they should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not compatible with newer versions of STL, and crashes on standard headers. Also, any headers that are not absolutely necessary to define the interface to Python should be removed if possible, that does make life a lot easier, even with SWIG, because it makes wrappers smaller. Best Regards, Alex PS: If you want I could give you benchmark results of cppyy vs cpyext+SWIG when we get it running, probably it will be of some interest. I could give you the entire code, but it would not be much use since it requires a ton of other libraries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From faassen at startifact.com Thu May 31 10:59:56 2012 From: faassen at startifact.com (Martijn Faassen) Date: Thu, 31 May 2012 10:59:56 +0200 Subject: [pypy-dev] change of strategy for the py3k branch? In-Reply-To: References: <4FC5D552.9050002@gmail.com> Message-ID: Hey, > Can you describe what sort of semantics you have in mind? Sure, I've discussed them before. The goal would be to have a Python 3 based project and use Python 2 modules/packages, or the other way around. That way it should become much easier to adopt Python 3, even in existing projects. To this end you'd need to have import magic that'd go across Pythons: # in python 3 foo = python2_import(''foo') # in python 2 bar = python3_import('bar') These would import the modules in the appropriate interpreter, and then wrap them in such a way that they become usable from the other interpreter. Later, you could come up with more sophisticated ways to designate a module "python 2" or "python 3" so that the normal 'import' statement will do something equivalent to the above (if you happen to know there are no namespace conflicts). >Would you like to have two copies of builtin modules? Yes, a separate copy for each interpreter. > How about namespaces? Yes, module name spaces should be separate. If you want to make a Python 2 library available in Python 3 you can use the import magic to do so. > What about objects being passed from one interpreter to the another? Would they magically > change or would they be "py2k dict" and "py3k dict"? If you can describe the > semantics of a proposed beast I'm willing to answer how likely it is to > happen They would be wrapped. I understand PyPy supports perfect proxies (I've seen the network-based demonstration). So you'd wrap a Python 3 object in a Python 2 wrapper, and vice versa. So a Python 3 proxy for a Python 2 object would: * make sure any attribute accesses are translated to Python 3 objects. (for immutables, a straight conversion is enough, otherwise a proxy) * a method proxy would make sure that any arguments are proxied from Python 3 to Python 2 (or straight conversion in case of an immutable if that'd be faster. Or a proxy unwrapping in case you are dealing with a Python 2 to 3 proxy already), and any return values are proxied from Python 2 to Python 3. The proxies for various built-ins such as dict would of course make sure that method calls are translated. You need to able to be able to declare various things about arguments and return values in some tricky cases like where a Python 2 string is involved; is it to be interpreted as a Python 3 string or a Python 3 bytes? Declarations could go into a central registry that is consulted by the proxy-ing mechanism, we can come up with nicer syntax later. The idea is that you could make declarations about a Python 2 library externally so you can use it within a Python 3 context. One way to think about this is a FFI from Python to Python. You'd need Python 2 to 3 proxies, Python 3 to 2 proxies, and various proxy wrapping and unwrapping rules. Regards, Martijn From holger at merlinux.eu Thu May 31 11:12:17 2012 From: holger at merlinux.eu (holger krekel) Date: Thu, 31 May 2012 09:12:17 +0000 Subject: [pypy-dev] connecting multiple interpreters (was: Re: change of strategy for the py3k branch?) In-Reply-To: References: <4FC5D552.9050002@gmail.com> Message-ID: <20120531091217.GI10174@merlinux.eu> Hi Martijn, On Wed, May 30, 2012 at 19:24 +0200, Martijn Faassen wrote: > Hi there, > > Just throwing in my little bit: any change that is made that would > make it easier to run Python 2 and Python 3 interpretors in the same > process would interesting, as I'm still vaguely dreaming (nothing > more) of a combined interpreter that can run both Python 2 and Python > 3 code. Is there a strong reason you want this in the same process? If not you might look into using execnet [1] for connecting python2 and python3 interpreters which then run in two separate processes. One can build something higher level on top of the base execnet communication along the lines of your "python3_import" suggestion. It seems you anyway need largely disconnected interpreter states. On a sidenote, Quora uses execnet to connect python2 and PyPy [2]. best, holger [1] http://codespeak.net/execnet/example/hybridpython.html [2] http://www.quora.com/Quora-Infrastructure/Did-Quoras-switch-to-PyPy-result-in-increased-memory-consumption > Regards, > > Martijn > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From faassen at startifact.com Thu May 31 11:27:44 2012 From: faassen at startifact.com (Martijn Faassen) Date: Thu, 31 May 2012 11:27:44 +0200 Subject: [pypy-dev] connecting multiple interpreters (was: Re: change of strategy for the py3k branch?) In-Reply-To: <20120531091217.GI10174@merlinux.eu> References: <4FC5D552.9050002@gmail.com> <20120531091217.GI10174@merlinux.eu> Message-ID: On Thu, May 31, 2012 at 11:12 AM, holger krekel wrote: > Hi Martijn, > > On Wed, May 30, 2012 at 19:24 +0200, Martijn Faassen wrote: >> Hi there, >> >> Just throwing in my little bit: any change that is made that would >> make it easier to run Python 2 and Python 3 interpretors in the same >> process would interesting, as I'm still vaguely dreaming (nothing >> more) of a combined interpreter that can run both Python 2 and Python >> 3 code. > > Is there a strong reason you want this in the same process? > > If not you might look into using execnet [1] for connecting python2 and > python3 interpreters which then run in two separate processes. ?One can > build something higher level on top of the base execnet communication > along the lines of your "python3_import" suggestion. ?It seems you anyway > need largely disconnected interpreter states. That's an interesting idea. I don't think it would accomplish exactly the same goals though. I can see two reasons not to do so: * developer simplicity: you just start up your Python 3 as usual, you can now use Python 2 modules in your project. No need to come up with a networked system. * efficiency: if you do a lot of calls using a Python 2 library in a Python 3 project, you'd like this to work pretty quickly. I think using in-process proxies can be made to be quite inexpensive. I think a networked approach is useful if you want communicating applications, but I'm talking more about a single application that uses libraries that might be written in another language. That is why I like the FFI analogy; when you interface with a C library from Python you generally also wouldn't want to use a networked approach. You *would* do this to interface with a networked application written in C. To use libxml2 in Python I'd use lxml. If there's already a C-based web service that uses libxml2 I'd use that service, but that's a different situation. I wouldn't want to have to *have* to do this just to be able to use libxml2. I think execnet blurs the line between library and application integration somewhat, as it allows very intimately communicating applications, but isn't the line still there? Regards, Martijn From alex.gaynor at gmail.com Thu May 31 22:28:01 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 31 May 2012 15:28:01 -0500 Subject: [pypy-dev] PyPy PPA Message-ID: Hey all, I'm hoping the people behind https://launchpad.net/~pypy/+archive/ppa are hear :) I'm wondering what the status of this (or any other PPAs) are, and whether it'd be possible to get a PPA that works with Ubuntu 12.04. The people behind http://travis-ci.org/ which is an awesome CI service, would like to be able to deploy PyPy via a deb on 12.04. Thanks, Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From gary.poster at canonical.com Thu May 31 22:38:16 2012 From: gary.poster at canonical.com (Gary Poster) Date: Thu, 31 May 2012 16:38:16 -0400 Subject: [pypy-dev] PyPy PPA In-Reply-To: References: Message-ID: <4FC7D6B8.3030306@canonical.com> On 05/31/2012 04:28 PM, Alex Gaynor wrote: > Hey all, > > I'm hoping the people > behind https://launchpad.net/~pypy/+archive/ppa are hear :) I'm > wondering what the status of this (or any other PPAs) are, and whether > it'd be possible to get a PPA that works with Ubuntu 12.04. The people > behind http://travis-ci.org/ which is an awesome CI service, would like > to be able to deploy PyPy via a deb on 12.04. PyPy is available in Precise, in universe, if that helps. Version: 1.8+dfsg-2 Gary From holger at merlinux.eu Thu May 31 23:18:09 2012 From: holger at merlinux.eu (holger krekel) Date: Thu, 31 May 2012 21:18:09 +0000 Subject: [pypy-dev] connecting multiple interpreters (was: Re: change of strategy for the py3k branch?) In-Reply-To: References: <4FC5D552.9050002@gmail.com> <20120531091217.GI10174@merlinux.eu> Message-ID: <20120531211809.GP10174@merlinux.eu> On Thu, May 31, 2012 at 11:27 +0200, Martijn Faassen wrote: > On Thu, May 31, 2012 at 11:12 AM, holger krekel wrote: > > On Wed, May 30, 2012 at 19:24 +0200, Martijn Faassen wrote: > >> Just throwing in my little bit: any change that is made that would > >> make it easier to run Python 2 and Python 3 interpretors in the same > >> process would interesting, as I'm still vaguely dreaming (nothing > >> more) of a combined interpreter that can run both Python 2 and Python > >> 3 code. > > > > Is there a strong reason you want this in the same process? > > > > If not you might look into using execnet [1] for connecting python2 and > > python3 interpreters which then run in two separate processes. ?One can > > build something higher level on top of the base execnet communication > > along the lines of your "python3_import" suggestion. ?It seems you anyway > > need largely disconnected interpreter states. > > That's an interesting idea. I don't think it would accomplish exactly > the same goals though. > > I can see two reasons not to do so: > > * developer simplicity: you just start up your Python 3 as usual, you > can now use Python 2 modules in your project. No need to come up with > a networked system. not sure i understand what you mean with "networked" system here. With gw = execnet.makegateway("python3") a subprocess is created running with python3. It's true that the channel send/receive uses a network metapher but underlying is process-to-process communication, no network involved. > * efficiency: if you do a lot of calls using a Python 2 library in a > Python 3 project, you'd like this to work pretty quickly. I think > using in-process proxies can be made to be quite inexpensive. it all depends i guess. It seems that for Quora it was fast enough to call from PyPy into several libraries deployed on cpython. Moreover, if you need to munge data coming out from library function calls the proxy approach may require a lot of communication between the two interpreters and even if this happens in-process it is overhead. With execnet you can execute the munging code with the interpreter running the library and only send back the result you need. (On a side note, with a proxy approach you also need to carefully design lifecycle/GC issues for out-of-interpreter references). > I think a networked approach is useful if you want communicating > applications, but I'm talking more about a single application that > uses libraries that might be written in another language. That is why > I like the FFI analogy; when you interface with a C library from > Python you generally also wouldn't want to use a networked approach. > You *would* do this to interface with a networked application written > in C. To use libxml2 in Python I'd use lxml. If there's already a > C-based web service that uses libxml2 I'd use that service, but that's > a different situation. I wouldn't want to have to *have* to do this > just to be able to use libxml2. > > I think execnet blurs the line between library and application > integration somewhat, as it allows very intimately communicating > applications, but isn't the line still there? There is a line, sure. It is blurred because one side can send code to the other. Which makes a difference in a similar way how sending Javascript to the client makes a difference - it reduces communication overhead and makes things faster on the client side. To conclude, i wouldn't be overly concerned by process-to-subprocess communication costs. If i had to combine py3 and py2 code (throw PyPy in to your likening) i'd go down the Quora route and see how far it carries. After all, this is only an intermediate solution until everthing happily runs on Python3 anyway, right? ;) best, holger