From zariko.taba at gmail.com Fri Jul 1 14:41:56 2011 From: zariko.taba at gmail.com (Zariko Taba) Date: Fri, 1 Jul 2011 14:41:56 +0200 Subject: [pypy-dev] Floating point computation Message-ID: Hi pypy list, I'd like to prototype an interpreter in rpython. But this interpreter must feature floating point support. Something like the context in the decimal module of python : (decimal is basically a floating point unit in python) import decimal # do an operation decimal.Decimal("6e4565465") * decimal.Decimal("6e67654") # Verify overflow decimal.getcontext().flags[decimal.Overflow] # leads 1 => overflow ! My interpreter must provide access to the "flags" field (OverFlow, Inexact, DivisionByZero...) Currently I have a prototype in c and I directly use the intrinsics of X86 (sse2) to retrieve theses flags : // compute a sqrt using intrinsics __builtin_ia32_sqrtss(...); // get status register of sse2 fpu int mxcsr = __builtin_ia32_stmxcsr(); // extract overflow bit : overflow = (mxcsr & 0x8); // extract ... If I want to implement this in rpython, I see 2 solutions : 1. Porting decimal to rpython : seems to be a lot of work and I wonder about the performances. 2. Wrapping floating in a c library and use ctype : code clarity and jit integration will be bad. Do you see another possibility ? Any advices on this ? Thanks ! Zariko. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri Jul 1 14:54:25 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 1 Jul 2011 14:54:25 +0200 Subject: [pypy-dev] Floating point computation In-Reply-To: References: Message-ID: Hi, On Fri, Jul 1, 2011 at 2:41 PM, Zariko Taba wrote: > // compute a sqrt using intrinsics > __builtin_ia32_sqrtss(...); > // get status register of sse2 fpu > int mxcsr = __builtin_ia32_stmxcsr(); > // extract overflow bit : > overflow = (mxcsr & 0x8); > // extract ... There is a third solution: do exactly the same kind of code, but in RPython instead of in C. This means using the rffi module. For example: builtin_ia32_sqrtss = rffi.llexternal("__builtin_ia32_sqrtss", [argtypes...], restype, _nowrapper=True, _callable=emulate_ia32_sqrtss) def f(..): ... builtin_ia32_sqrtss(..) ... with emulate_ia32_sqrtss() being a pure Python function that is going to be called only during tests. After translation to C, it will just turn into the C code that you posted above. For more examples, grep for "llexternal" in pypy/rlib/. A bient?t, Armin. From fijall at gmail.com Fri Jul 1 14:57:38 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 1 Jul 2011 14:57:38 +0200 Subject: [pypy-dev] Floating point computation In-Reply-To: References: Message-ID: On Fri, Jul 1, 2011 at 2:54 PM, Armin Rigo wrote: > Hi, > > On Fri, Jul 1, 2011 at 2:41 PM, Zariko Taba wrote: >> // compute a sqrt using intrinsics >> __builtin_ia32_sqrtss(...); >> // get status register of sse2 fpu >> int mxcsr = __builtin_ia32_stmxcsr(); >> // extract overflow bit : >> overflow = (mxcsr & 0x8); >> // extract ... > > There is a third solution: do exactly the same kind of code, but in > RPython instead of in C. ?This means using the rffi module. ?For > example: > > builtin_ia32_sqrtss = rffi.llexternal("__builtin_ia32_sqrtss", [argtypes...], > ? ?restype, _nowrapper=True, _callable=emulate_ia32_sqrtss) > > def f(..): > ? ?... > ? ?builtin_ia32_sqrtss(..) > ? ?... > > with emulate_ia32_sqrtss() being a pure Python function that is going > to be called only during tests. ?After translation to C, it will just > turn into the C code that you posted above. ?For more examples, grep > for "llexternal" in pypy/rlib/. But it won't become an assembler instruction in the JIT, it'll still be a call. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From arigo at tunes.org Fri Jul 1 15:24:44 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 1 Jul 2011 15:24:44 +0200 Subject: [pypy-dev] Floating point computation In-Reply-To: References: Message-ID: Hi, On Fri, Jul 1, 2011 at 2:57 PM, Maciej Fijalkowski wrote: > But it won't become an assembler instruction in the JIT, it'll still be a call. Indeed. But fixing this is not too hard. Grep for "math_sqrt" and "MATH_SQRT" in pypy/jit/*/*.py for an example of how we turn sqrt_nonneg(x) into a single assembler instruction. The function sqrt_nonneg() is defined in pypy/rpython/lltypesystem/module/ll_math.py, and is (indirectly) used when we call math.sqrt() in RPython. Of course if the idea introduced by math_sqrt is going to be used for a lot of other functions, like the ones you'll need to introduce, then we can also think of a general solution that would do mostly everything for you with just a keyword argument to llexternal(). A bient?t, Armin. From zariko.taba at gmail.com Fri Jul 1 17:08:32 2011 From: zariko.taba at gmail.com (Zariko Taba) Date: Fri, 1 Jul 2011 17:08:32 +0200 Subject: [pypy-dev] Floating point computation In-Reply-To: References: Message-ID: On Fri, Jul 1, 2011 at 2:54 PM, Armin Rigo wrote: > > There is a third solution: do exactly the same kind of code, but in > RPython instead of in C. This means using the rffi module. For > example: > > builtin_ia32_sqrtss = rffi.llexternal("__builtin_ia32_sqrtss", > [argtypes...], > restype, _nowrapper=True, _callable=emulate_ia32_sqrtss) > Awesome ! I will give it a try. :) Thanks ! Zariko. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Fri Jul 1 18:40:57 2011 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 1 Jul 2011 09:40:57 -0700 (PDT) Subject: [pypy-dev] stuck with translation error Message-ID: Hi, ran into this error, which I haven't seen before and which I can't pin down: [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 254, in specialize_more_blocks [translation:ERROR] self.specialize_block(block) [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 406, in specialize_block [translation:ERROR] self.translate_hl_to_ll(hop, varmapping) [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 535, in translate_hl_to_ll [translation:ERROR] resultvar = hop.dispatch() [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 768, in dispatch [translation:ERROR] return translate_meth(self) [translation:ERROR] File "<487-codegen /home/wlav/pypydev/pypy/pypy/rpython/rtyper.py:610>", line 4, in translate_op_simple_call [translation:ERROR] return r_arg1.rtype_simple_call(hop) [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rpbc.py", line 723, in rtype_simple_call [translation:ERROR] return self.redispatch_call(hop, call_args=False) [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rpbc.py", line 772, in redispatch_call [translation:ERROR] assert hop.nb_args == 1, ("arguments passed to __init__, " [translation:ERROR] AssertionError: arguments passed to __init__, but no __init__! I don't find anything that could be related to local code, until I go up to specialize_block. That block has: (Pdb+) block.operations [v1 = call_args((function free), ((1, ('flavor', 'track_... False)), v0, ('raw'), (False)), v2 = simple_call((type error), ('out of resources'))] which I presume is lltype.free() of which I have a few calls, but no recent changes (and I don't understand the complaint about __init__ at that point). I'm going to roll back history to see where things broke, but if the above rings a bell with someone, that could spare me some time ... Thanks! Cheers, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From amauryfa at gmail.com Fri Jul 1 19:43:30 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 1 Jul 2011 19:43:30 +0200 Subject: [pypy-dev] stuck with translation error In-Reply-To: References: Message-ID: Hi, 2011/7/1 : > [translation:ERROR] ? ?File > "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 254, in specialize_more_blocks > [translation:ERROR] ? ? self.specialize_block(block) > [translation:ERROR] ? ?File "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 406, in specialize_block > [translation:ERROR] ? ? self.translate_hl_to_ll(hop, varmapping) > [translation:ERROR] ? ?File "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 535, in translate_hl_to_ll > [translation:ERROR] ? ? resultvar = hop.dispatch() > [translation:ERROR] ? ?File "/home/wlav/pypydev/pypy/pypy/rpython/rtyper.py", line 768, in dispatch > [translation:ERROR] ? ? return translate_meth(self) > [translation:ERROR] ? ?File "<487-codegen /home/wlav/pypydev/pypy/pypy/rpython/rtyper.py:610>", line 4, in translate_op_simple_call > [translation:ERROR] ? ? return r_arg1.rtype_simple_call(hop) > [translation:ERROR] ? ?File "/home/wlav/pypydev/pypy/pypy/rpython/rpbc.py", line 723, in rtype_simple_call > [translation:ERROR] ? ? return self.redispatch_call(hop, call_args=False) > [translation:ERROR] ? ?File "/home/wlav/pypydev/pypy/pypy/rpython/rpbc.py", line 772, in redispatch_call > [translation:ERROR] ? ? assert hop.nb_args == 1, ("arguments passed to __init__, " > [translation:ERROR] ?AssertionError: arguments passed to __init__, but no __init__! > > I don't find anything that could be related to local code, until I go up > to specialize_block. That block has: > > (Pdb+) block.operations > [v1 = call_args((function free), ((1, ('flavor', 'track_... False)), v0, ('raw'), (False)), v2 = simple_call((type error), ('out of resources'))] > > which I presume is lltype.free() of which I have a few calls, but no recent > changes (and I don't understand the complaint about __init__ at that point). > > I'm going to roll back history to see where things broke, but if the above > rings a bell with someone, that could spare me some time ... Thanks! Hm, it can also be the next operation: v2 = simple_call((type error), ('out of resources')) What is this "error" object? Some local exception class? -- Amaury Forgeot d'Arc From wlavrijsen at lbl.gov Fri Jul 1 20:04:19 2011 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 1 Jul 2011 11:04:19 -0700 (PDT) Subject: [pypy-dev] stuck with translation error In-Reply-To: References: Message-ID: Hi Amaury, >> [v1 = call_args((function free), ((1, ('flavor', 'track_... False)), v0, ('raw'), (False)), v2 = simple_call((type error), ('out of resources'))] > > Hm, it can also be the next operation: v2 = simple_call((type error), > ('out of resources')) I was under the impression that that was a standard error, but then again that would have been for malloc(), not free(). I searched again and came up with is code in ll_thread.py: def allocate_ll_lock(): # track_allocation=False here; be careful to lltype.free() it. The # reason it is set to False is that we get it from all app-level # lock objects, as well as from the GIL, which exists at shutdown. ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw', track_allocation=False) res = c_thread_lock_init(ll_lock) if rffi.cast(lltype.Signed, res) <= 0: lltype.free(ll_lock, flavor='raw', track_allocation=False) raise error("out of resources") return ll_lock which looks very much like it (the link to the exit is a raise). So I think that it isn't my code that doesn't translate. :) Beats me then, why it would bite me. > What is this "error" object? Some local exception class? I can look again in a half an hour: I had just restarted the translation after a removing some of my recent changes. But I'll bet it's what I see here above in allocate_ll_lock. The module ll_thread.py defines: class error(Exception): pass Thanks, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From anto.cuni at gmail.com Sat Jul 2 11:59:40 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 02 Jul 2011 11:59:40 +0200 Subject: [pypy-dev] bounties for pypy In-Reply-To: References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> Message-ID: <4E0EEC0C.5010809@gmail.com> Hello Ian, On 29/06/11 15:16, Ian Ozsvald wrote: > Ian. > ps. I posted the v0.1 PDF of my High Performance Python tutorial this > morning (it is based on my EuroPython training session). It has a > section on PyPy and I'd happily accept input if that section should be > expanded:http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/ I read your tutorial and tried some of the code, nice work. I tried to use the "bettermath" approach also on the pure python code, and run it on PyPy: on my machine, it takes 2.4 seconds instead of 6.2. For comparison, cython with bettermath takes 0.57 seconds, i.e. it's about 4 times faster than PyPy (with a trunk version of PyPy, did not try with pypy 1.5). For this kind of code, there is no fundamental reason why PyPy should be slower than cython, so I'll investigate a bit to see what is the problem. But still, I think that you should mention this "bettermath" implementation which is much faster on PyPy. ciao, Anto From ian at ianozsvald.com Sun Jul 3 12:03:14 2011 From: ian at ianozsvald.com (Ian Ozsvald) Date: Sun, 3 Jul 2011 11:03:14 +0100 Subject: [pypy-dev] bounties for pypy In-Reply-To: <4E0EEC0C.5010809@gmail.com> References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: Cool :-) Thanks Antonio. I'd left a note to myself in the report because I figured expanding the math to the primitive operations might help PyPy (seeing as it helped Cython and ShedSkin too). I'll redo the timings and add them to the v0.2 report (probably in two weeks time - I'm waiting for feedback from several people). I've already tried a trunk version of PyPy and it was faster than PyPy, I'll take whichever build is the most recent and replace PyPy1.5 for the next updates to the report. If there is a trick to e.g. re-ordering the operations so it runs faster in PyPy, I'd be happy to accept a modification. Hopefully there will also be a pyOpenCL version to add to the pyCUDA examples by then. Much obliged, Ian. On 2 July 2011 10:59, Antonio Cuni wrote: > Hello Ian, > > On 29/06/11 15:16, Ian Ozsvald wrote: >> >> Ian. >> ps. I posted the v0.1 PDF of my High Performance Python tutorial this >> morning (it is based on my EuroPython training session). It has a >> section on PyPy and I'd happily accept input if that section should be >> >> expanded:http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/ > > I read your tutorial and tried some of the code, nice work. > > I tried to use the "bettermath" approach also on the pure python code, and > run it on PyPy: on my machine, it takes 2.4 seconds instead of 6.2. > For comparison, cython with bettermath takes 0.57 seconds, i.e. it's about 4 > times faster than PyPy (with a trunk version of PyPy, did not try with pypy > 1.5). > > For this kind of code, there is no fundamental reason why PyPy should be > slower than cython, so I'll investigate a bit to see what is the problem. > ?But still, I think that you should mention this "bettermath" implementation > which is much faster on PyPy. > > ciao, > Anto > -- Ian Ozsvald (A.I. researcher, screencaster) ian at IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald From fijall at gmail.com Sun Jul 3 19:34:53 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 3 Jul 2011 19:34:53 +0200 Subject: [pypy-dev] bounties for pypy In-Reply-To: References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: On Sun, Jul 3, 2011 at 12:03 PM, Ian Ozsvald wrote: > Cool :-) Thanks Antonio. I'd left a note to myself in the report > because I figured expanding the math to the primitive operations might > help PyPy (seeing as it helped Cython and ShedSkin too). I'll redo the > timings and add them to the v0.2 report (probably in two weeks time - > I'm waiting for feedback from several people). > > I've already tried a trunk version of PyPy and it was faster than > PyPy, I'll take whichever build is the most recent and replace PyPy1.5 > for the next updates to the report. If there is a trick to e.g. > re-ordering the operations so it runs faster in PyPy, I'd be happy to > accept a modification. > You can also use numpy arrays (or array from array module) on PyPy to get even more speedups. It's annoying because as of now pypy doesn't support much of numpy (1D float arrays only with ops please :) but it's a good start From fijall at gmail.com Sun Jul 3 19:38:32 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 3 Jul 2011 19:38:32 +0200 Subject: [pypy-dev] bounties for pypy In-Reply-To: References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: On Sun, Jul 3, 2011 at 7:34 PM, Maciej Fijalkowski wrote: > On Sun, Jul 3, 2011 at 12:03 PM, Ian Ozsvald wrote: >> Cool :-) Thanks Antonio. I'd left a note to myself in the report >> because I figured expanding the math to the primitive operations might >> help PyPy (seeing as it helped Cython and ShedSkin too). I'll redo the >> timings and add them to the v0.2 report (probably in two weeks time - >> I'm waiting for feedback from several people). >> >> I've already tried a trunk version of PyPy and it was faster than >> PyPy, I'll take whichever build is the most recent and replace PyPy1.5 >> for the next updates to the report. If there is a trick to e.g. >> re-ordering the operations so it runs faster in PyPy, I'd be happy to >> accept a modification. >> > > You can also use numpy arrays (or array from array module) on PyPy to > get even more speedups. It's annoying because as of now pypy doesn't > support much of numpy (1D float arrays only with ops please :) but > it's a good start > "During 2011 at least it looks as though numpy integration will not happen." I personally challenge this statement :) From ian at ianozsvald.com Sun Jul 3 20:35:10 2011 From: ian at ianozsvald.com (Ian Ozsvald) Date: Sun, 3 Jul 2011 19:35:10 +0100 Subject: [pypy-dev] bounties for pypy In-Reply-To: References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: Hi Maciej. I said that based on what was discussed in Armin+Antonio's talk at EuroPython. If it takes 6+months for a numpy re-implementation then I'm suggesting that we're into 2012. I'd be *very* happy to be proved wrong! Do Python 'array' objects run faster than lists in PyPy? I believe that in CPython they run at the same speed (i.e. they're just a convenient storage system, they don't offer any of numpy's efficient math benefits). Does the micronumpy library support doubles yet? If so I'd be happy to give it a go. If 'micronumpy' is the wrong name then tell me where I should look, I'm just going on what I've remembered from the PyPy-blog discussions about numpy support. Cheers, Ian. On 3 July 2011 18:38, Maciej Fijalkowski wrote: > On Sun, Jul 3, 2011 at 7:34 PM, Maciej Fijalkowski wrote: >> On Sun, Jul 3, 2011 at 12:03 PM, Ian Ozsvald wrote: >>> Cool :-) Thanks Antonio. I'd left a note to myself in the report >>> because I figured expanding the math to the primitive operations might >>> help PyPy (seeing as it helped Cython and ShedSkin too). I'll redo the >>> timings and add them to the v0.2 report (probably in two weeks time - >>> I'm waiting for feedback from several people). >>> >>> I've already tried a trunk version of PyPy and it was faster than >>> PyPy, I'll take whichever build is the most recent and replace PyPy1.5 >>> for the next updates to the report. If there is a trick to e.g. >>> re-ordering the operations so it runs faster in PyPy, I'd be happy to >>> accept a modification. >>> >> >> You can also use numpy arrays (or array from array module) on PyPy to >> get even more speedups. It's annoying because as of now pypy doesn't >> support much of numpy (1D float arrays only with ops please :) but >> it's a good start >> > > "During 2011 at least it looks as > though numpy integration will not happen." I personally challenge this > statement :) > -- Ian Ozsvald (A.I. researcher, screencaster) ian at IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald From fijall at gmail.com Sun Jul 3 20:54:59 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 3 Jul 2011 20:54:59 +0200 Subject: [pypy-dev] bounties for pypy In-Reply-To: References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: On Sun, Jul 3, 2011 at 8:35 PM, Ian Ozsvald wrote: > Hi Maciej. I said that based on what was discussed in Armin+Antonio's > talk at EuroPython. If it takes 6+months for a numpy re-implementation > then I'm suggesting that we're into 2012. I'd be *very* happy to be > proved wrong! I would estimate numpy for less than 6 months (numpy, not scipy and not matplotlib), but well that's me. There is also a lot of outside contribution, so it might take less wall time than manmonth time. From my perspective, if we can secure *some* funding for numpy, it should be ready sooner than 6 months from now. > > Do Python 'array' objects run faster than lists in PyPy? I believe > that in CPython they run at the same speed (i.e. they're just a > convenient storage system, they don't offer any of numpy's efficient > math benefits). The situation between PyPy and CPython is quite different here. As of now, both PyPy and CPython store wrapped objects in lists (let's call them PyIntObject) and unwrapped (C level int) in numpy arrays. In case you have only interpreter, when reading a list you do: py_x = list[index] do_something_with_py_x when reading from numpy array (or array.array), you do: py_x = new PyIntObject(array[index]) do_something_with_py_x so you use less memory (cache is better), but you allocate a new object each iteration (bad) But in case of the JIT, what happens is (list case) py_x = list[index] x = py_x.value do_something_with_x # x as in integer value not py_x array: x = list[index] do_something_with_x Hence, no allocation and memory saving - huge win! > > Does the micronumpy library support doubles yet? If so I'd be happy to > give it a go. If 'micronumpy' is the wrong name then tell me where I > should look, I'm just going on what I've remembered from the PyPy-blog > discussions about numpy support. yes, doubles is pretty much all it supports as of now :) micronumpy is a good name, but it comes under a name "numpy", so you can do "import numpy". Not much will work though :-) Note that we're also in the process of implementing faster vectorized operations, like numpyexpr, except better. The list of operations is very limited as of now though. Cheers, fijal PS. If it's unclear, I can try to explain more or pop up on IRC. From ian at ianozsvald.com Mon Jul 4 11:40:40 2011 From: ian at ianozsvald.com (Ian Ozsvald) Date: Mon, 4 Jul 2011 10:40:40 +0100 Subject: [pypy-dev] bounties for pypy In-Reply-To: References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: Re. micronumpy - all I'm doing in the Mandelbrot demo is multiplying and addition on doubles - that'll work now, right? If so I'll make the modification in a week or so when I next have some time. I'm guessing that there is no 'complex' support yet, even for basic operations? Adding micronumpy support would make for a nice addition to the v0.2 doc :-) Ian. re. the 6 month 'proper numpy' estimate - I'm just quoting some of the figures that were thrown around (with wide error margins depending on money raised, availability etc). I would *love* to see support come through sooner! On 3 July 2011 19:54, Maciej Fijalkowski wrote: > On Sun, Jul 3, 2011 at 8:35 PM, Ian Ozsvald wrote: >> Hi Maciej. I said that based on what was discussed in Armin+Antonio's >> talk at EuroPython. If it takes 6+months for a numpy re-implementation >> then I'm suggesting that we're into 2012. I'd be *very* happy to be >> proved wrong! > > I would estimate numpy for less than 6 months (numpy, not scipy and > not matplotlib), but well that's me. There is also a lot of outside > contribution, so it might take less wall time than manmonth time. From > my perspective, if we can secure *some* funding for numpy, it should > be ready sooner than 6 months from now. > >> >> Do Python 'array' objects run faster than lists in PyPy? I believe >> that in CPython they run at the same speed (i.e. they're just a >> convenient storage system, they don't offer any of numpy's efficient >> math benefits). > > The situation between PyPy and CPython is quite different here. As of > now, both PyPy and CPython store wrapped objects in lists (let's call > them PyIntObject) and unwrapped (C level int) in numpy arrays. In case > you have only interpreter, when reading a list you do: > > py_x = list[index] > do_something_with_py_x > > when reading from numpy array (or array.array), you do: > > py_x = new PyIntObject(array[index]) > do_something_with_py_x > > so you use less memory (cache is better), but you allocate a new > object each iteration (bad) > > But in case of the JIT, what happens is (list case) > > py_x = list[index] > x = py_x.value > do_something_with_x # x as in integer value not py_x > > array: > > x = list[index] > do_something_with_x > > Hence, no allocation and memory saving - huge win! > >> >> Does the micronumpy library support doubles yet? If so I'd be happy to >> give it a go. If 'micronumpy' is the wrong name then tell me where I >> should look, I'm just going on what I've remembered from the PyPy-blog >> discussions about numpy support. > > yes, doubles is pretty much all it supports as of now :) micronumpy is > a good name, but it comes under a name "numpy", so you can do "import > numpy". Not much will work though :-) > > Note that we're also in the process of implementing faster vectorized > operations, like numpyexpr, except better. The list of operations is > very limited as of now though. > > Cheers, > fijal > > PS. If it's unclear, I can try to explain more or pop up on IRC. > -- Ian Ozsvald (A.I. researcher, screencaster) ian at IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald From alex.gaynor at gmail.com Mon Jul 4 18:11:53 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 4 Jul 2011 09:11:53 -0700 Subject: [pypy-dev] Fwd: Speed.Python.org In-Reply-To: References: Message-ID: The following message is forwarded on behalf of Jesse Noller: Now that we have the machine, we need to start working on collecting/organizing the resources needed to get a shared codespeed system in place. After speaking with various people, we felt that overloading codespeed-dev, pypy-dev or python-dev with the discussions around this would be sub optimal. I've spun up a new mailing list here: http://mail.python.org/mailman/listinfo/speed Those who are interested in working on or contributing to the speed.python.org project can subscribe there. I personally can not lead the project, and so I will be looking to the current speed.pypy.org team, and python-dev contributors for leadership in this. I got you the hardware and hosting! :) jesse -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From romain.py at gmail.com Wed Jul 6 05:10:35 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Wed, 6 Jul 2011 05:10:35 +0200 Subject: [pypy-dev] Cython backend aiming PyPy Status Message-ID: <20110706031034.GA31841@ubuntu> Hi I created a blog post summarizing what I've done the last few weeks on the Cython backend aiming PyPy. It's located at this URL : http://rguillebert.blogspot.com/2011/07/cython-backend-aiming-pypy-status.html Cheers Romain From giuott at gmail.com Wed Jul 6 10:55:04 2011 From: giuott at gmail.com (Giuseppe Ottaviano) Date: Wed, 6 Jul 2011 09:55:04 +0100 Subject: [pypy-dev] algorithm used for float -> str conversion In-Reply-To: References: <201106300926.50520.alexandre.fayolle@logilab.fr> Message-ID: > Note that cpython2.7 (and pypy 1.5) already uses a specific algorithm > to convert float to strings: a slightly customized version of David Gay's > dtoa.c: http://www.netlib.org/fp/dtoa.c > it is already faster and more accurate than many libc implementations. The article says "the Grisu family acts as the default rendering algorithms in both the V8 and Mozilla Javascript engines (replacing David Gay's 17-year-old dtoa code)" I wonder if they had specific reasons to switch that could be relevant to pypy and cpython as well. For example, besides speed, Grisu implementations look way simpler than Gay's code because the latter uses arbitrary precision integers (and it has to implement them from scratch). For this reason dtoa also needs dynamic allocations, while Grisu only does simple arithmetic on machine-word integers (if my understanding is correct, from a quick look at the paper and the source of both dtoa and grisu). From wlavrijsen at lbl.gov Thu Jul 7 00:27:07 2011 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 6 Jul 2011 15:27:07 -0700 (PDT) Subject: [pypy-dev] stuck with translation error In-Reply-To: References: Message-ID: Hi, >> [translation:ERROR] File "<487-codegen /home/wlav/pypydev/pypy/pypy/rpython/rtyper.py:610>", line 4, in translate_op_simple_call >> [translation:ERROR] return r_arg1.rtype_simple_call(hop) >> [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rpbc.py", line 723, in rtype_simple_call >> [translation:ERROR] return self.redispatch_call(hop, call_args=False) >> [translation:ERROR] File "/home/wlav/pypydev/pypy/pypy/rpython/rpbc.py", line 772, in redispatch_call >> [translation:ERROR] assert hop.nb_args == 1, ("arguments passed to __init__, " >> [translation:ERROR] AssertionError: arguments passed to __init__, but no __init__! got it, and funnily enough it even had something to do with an __init__ (but that wasn't the actual cause). I had some code to raise an exception in init if an attempt was made to instantiate an abstract C++ class. In that code, I did: except Exception, e: if e.match(self.space, self.space.w_AttributeError): whereas it should have been: except OperationError, e: if e.match(self.space, self.space.w_AttributeError): the point being that the code changes an AttributeError (failure to find a suitable method, i.e. a usable C++ constructor) into a TypeError (stating that the class is abstract). I'm actually surprised that the bad code made it past the rtyper, given that class Exception does not actually have a match() method. Either way, I don't think I deserved such a horrible error message for such a little coding mistake. :} But there's a good deal of irony here in that the code that caused the horrible error message on me is there to make an error in user code more clear. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From wlavrijsen at lbl.gov Thu Jul 7 03:24:37 2011 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 6 Jul 2011 18:24:37 -0700 (PDT) Subject: [pypy-dev] unable to translate with current head of default In-Reply-To: References: <4DDCAEF2.5040100@gmail.com> Message-ID: Hi Armin, to come back to the problem that I've had with the jit ... it's gone now with current default tip. No idea what's up, but I'm plenty happy with things now working again for me. The "heap" optimization does have a significant effect on my code, though, so it's good to have it. I broke the fast path during one of the many merges with default when I had to resolve differences (to be fixed next week?), but the current numbers are still excellent: $ ../../../translator/goal/pypy-cppyy-c-070511 bench1.py rebuilding bench1.exe ... warming up ... C++ reference uses 0.032s :::: cppyy interp cost: 0.523s ( 16x) :::: cppyy python cost: 0.997s ( 31x) :::: pycintex cost: 50.554s (1579x) So a factor of 50 improvement over the current CPython bindings, something that used to be (back in December) a factor of 10 "only." Of course, for a real comparison, I should backport cppyy to pypy as of Dec '10 to know which changes are to be attributed to what, but none of this is the final product, so for now it's simply good enough. :) Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From christian at jensenbox.com Thu Jul 7 04:14:57 2011 From: christian at jensenbox.com (Christian Jensen) Date: Wed, 6 Jul 2011 19:14:57 -0700 Subject: [pypy-dev] Postgres Message-ID: <-8985362543625951028@unknownmsgid> What is the best way to work with Postgres via Django? Is it still slow or have there been recent developments? Sent from my iPhone From benjamin at python.org Thu Jul 7 04:20:56 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 6 Jul 2011 21:20:56 -0500 Subject: [pypy-dev] Postgres In-Reply-To: <-8985362543625951028@unknownmsgid> References: <-8985362543625951028@unknownmsgid> Message-ID: https://bitbucket.org/alex_gaynor/pypy-postgresql/overview 2011/7/6 Christian Jensen : > What is the best way to work with Postgres via Django? Is it still > slow or have there been recent developments? > > Sent from my iPhone > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -- Regards, Benjamin From christian at jensenbox.com Thu Jul 7 06:44:32 2011 From: christian at jensenbox.com (Christian Jensen) Date: Wed, 6 Jul 2011 21:44:32 -0700 Subject: [pypy-dev] Postgres In-Reply-To: References: <-8985362543625951028@unknownmsgid> Message-ID: <-7620406074375308895@unknownmsgid> Thanks. Are there any benchmarks or anything reasonable to indicate the speed in relation to cpython? Sent from my iPhone On Jul 6, 2011, at 7:20 PM, Benjamin Peterson wrote: > https://bitbucket.org/alex_gaynor/pypy-postgresql/overview > > 2011/7/6 Christian Jensen : >> What is the best way to work with Postgres via Django? Is it still >> slow or have there been recent developments? >> >> Sent from my iPhone >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > > > > -- > Regards, > Benjamin From vsapre80 at gmail.com Thu Jul 7 10:47:51 2011 From: vsapre80 at gmail.com (Vishal) Date: Thu, 7 Jul 2011 14:17:51 +0530 Subject: [pypy-dev] Floating point computation In-Reply-To: References: Message-ID: Hello, I would like to bring to your notice this article about exposing SIMD instructions to high level languages. http://drdobbs.com/blogs/tools/230600043 The Translation and JIT backend for PyPy may be able to allow Python programmers to use SIMD instructions directly from Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jul 7 10:59:26 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 7 Jul 2011 10:59:26 +0200 Subject: [pypy-dev] Floating point computation In-Reply-To: References: Message-ID: On Thu, Jul 7, 2011 at 10:47 AM, Vishal wrote: > Hello, > I would like to bring to your notice this article about exposing SIMD > instructions to high level languages. > http://drdobbs.com/blogs/tools/230600043 > The Translation and JIT backend for PyPy may be able to allow Python > programmers to use SIMD instructions directly from Python. We kind of want to do that automatically for numpy operations. > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > From neppord at gmail.com Thu Jul 7 19:53:36 2011 From: neppord at gmail.com (Samuel Ytterbrink) Date: Thu, 7 Jul 2011 19:53:36 +0200 Subject: [pypy-dev] Argparser in RPython Message-ID: Hi very one! I'm doing some planing on writing a option/arg parser in RPython to add to the rlib. How ever I'm verry new to RPython. I would appreciate all input on how i should approach the task a head. Does people think that there is a BIG benefit to implement a dropin replacement for either the argparser from stdlib or the optparser from stdlib. Or should i look at the pypy app-level implementation and try to write something that is( simpler) and follow its criteria. At the moment i don't have nice work enviroment( waiting for a new computer) so i cant realy try out the nice translatorshell and all it's test code. But when i get it I'll start hacking of. In the meantime ill do research and some planing. I'm hoping to use the rlib parsing tool just for the benefit of one more project using it and by that have more code for people to look at for exampples to the parsing lib. Is this overkill? -- //Samuel (Neppord) Ytterbrink -------------- next part -------------- An HTML attachment was scrubbed... URL: From neppord at gmail.com Thu Jul 7 20:19:19 2011 From: neppord at gmail.com (Samuel Ytterbrink) Date: Thu, 7 Jul 2011 20:19:19 +0200 Subject: [pypy-dev] Argparser in RPython In-Reply-To: References: Message-ID: Please dissregard the rlib parsing idea, I was not thinking straight. Den 7 jul 2011 19.53 skrev "Samuel Ytterbrink" : > Hi very one! > > I'm doing some planing on writing a option/arg parser in RPython to add to > the rlib. How ever I'm verry new to RPython. I would appreciate all input on > how i should approach the task a head. > > Does people think that there is a BIG benefit to implement a dropin > replacement for either the argparser from stdlib or the optparser from > stdlib. > > Or should i look at the pypy app-level implementation and try to write > something that is( simpler) and follow its criteria. > > At the moment i don't have nice work enviroment( waiting for a new computer) > so i cant realy try out the nice translatorshell and all it's test code. But > when i get it I'll start hacking of. In the meantime ill do research and > some planing. > > I'm hoping to use the rlib parsing tool just for the benefit of one more > project using it and by that have more code for people to look at for > exampples to the parsing lib. Is this overkill? > > -- > //Samuel (Neppord) Ytterbrink -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandervpetrov at gmail.com Fri Jul 8 01:03:20 2011 From: alexandervpetrov at gmail.com (Alexander Petrov) Date: Fri, 8 Jul 2011 02:03:20 +0300 Subject: [pypy-dev] PyPy is much slower than CPython example / question Message-ID: Hi. I'm new to PyPy and was trying to run some tests to see orders of speed improvement. Short script generating list of prime numbers using rather straightforward implementation of Eratosthene's sieve. Script: http://paste.pocoo.org/show/432727/ Typical results: http://paste.pocoo.org/show/432733/ (I thought that is due to absense of SSE2 on first computer, but I've rechecked on Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with similar results). I'm getting that CPython is nearly 4-8 times faster than PyPy. Is it a bug in PyPy or what is wrong (may be "specific" to PyPy) in my script? Alex. From alex.gaynor at gmail.com Fri Jul 8 01:49:33 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 7 Jul 2011 16:49:33 -0700 Subject: [pypy-dev] Object identity and dict strategies Message-ID: Hi all, I've now spoken with several developers about the object identity that arises out of hte new dict strategies, and all seem to think that the current implementation breaks Python's semantics, that is: x = 3 z = {x: None} assert next(iter(z)) is x will fail. The only solution I see to this that maintains the correct semantics in all cases is to specialize is/id() for primitive types. That is to say, all integers of any given value have the same id() and x is y is true. Doing x is y is true is easy, you simply have a multimethod which dispatches on the types and compares intval for W_IntObjects (objects of differing types can never have the same identity) however the question is what to use for id() that is unique, never changes for an int, and doesn't intersect with any other live object. In particular the last constraint is the most difficult I think. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Fri Jul 8 02:12:59 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 7 Jul 2011 17:12:59 -0700 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: Message-ID: On Thu, Jul 7, 2011 at 4:03 PM, Alexander Petrov wrote: > Hi. > > I'm new to PyPy and was trying to run some tests to see orders of > speed improvement. > > Short script generating list of prime numbers using rather > straightforward implementation of Eratosthene's sieve. > Script: http://paste.pocoo.org/show/432727/ > > Typical results: http://paste.pocoo.org/show/432733/ > (I thought that is due to absense of SSE2 on first computer, but I've > rechecked on Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with similar > results). > > I'm getting that CPython is nearly 4-8 times faster than PyPy. > Is it a bug in PyPy or what is wrong (may be "specific" to PyPy) in my > script? > > Alex. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > I haven't dug into this too deeply to see where the slowness is, however if you replace the setslice + repeat thingy with `for i in range(--that expression--): primes[i] = False` it becomes significantly faster, PyPy is 10x faster, and ~2x faster the original CPython version (CPython is ~2x slower when it's written this way). Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From romain.py at gmail.com Fri Jul 8 02:30:48 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Fri, 8 Jul 2011 02:30:48 +0200 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: Message-ID: <20110708003048.GA8477@ubuntu> Hi When I change this line: primes[i*i:N+1:i] = repeat(False, len(primes[i*i:N+1:i])) into this : primes[i*i:N+1:i] = [False] * len(primes[i*i:N+1:i]) PyPy is much faster (but is still slower than CPython), so I would guess that the repeat function is the one to blame. Cheers Romain On Fri, Jul 08, 2011 at 02:03:20AM +0300, Alexander Petrov wrote: > Hi. > > I'm new to PyPy and was trying to run some tests to see orders of > speed improvement. > > Short script generating list of prime numbers using rather > straightforward implementation of Eratosthene's sieve. > Script: http://paste.pocoo.org/show/432727/ > > Typical results: http://paste.pocoo.org/show/432733/ > (I thought that is due to absense of SSE2 on first computer, but I've > rechecked on Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with similar > results). > > I'm getting that CPython is nearly 4-8 times faster than PyPy. > Is it a bug in PyPy or what is wrong (may be "specific" to PyPy) in my script? > > Alex. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From alex.gaynor at gmail.com Fri Jul 8 02:38:44 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 7 Jul 2011 17:38:44 -0700 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: <20110708003048.GA8477@ubuntu> References: <20110708003048.GA8477@ubuntu> Message-ID: repeat itself is not slow, it's just that when it's used it iterates over it, in RPython (meaning it's not jit'd) which results in a dictionary lookup for the next() method at every iteration, which is slowish, list hits a special case so it doesn' thave that overhead. Alex On Thu, Jul 7, 2011 at 5:30 PM, Romain Guillebert wrote: > Hi > > When I change this line: > > primes[i*i:N+1:i] = repeat(False, len(primes[i*i:N+1:i])) > > into this : > > primes[i*i:N+1:i] = [False] * len(primes[i*i:N+1:i]) > > PyPy is much faster (but is still slower than CPython), so I would guess > that the repeat function is the one to blame. > > Cheers > Romain > > On Fri, Jul 08, 2011 at 02:03:20AM +0300, Alexander Petrov wrote: > > Hi. > > > > I'm new to PyPy and was trying to run some tests to see orders of > > speed improvement. > > > > Short script generating list of prime numbers using rather > > straightforward implementation of Eratosthene's sieve. > > Script: http://paste.pocoo.org/show/432727/ > > > > Typical results: http://paste.pocoo.org/show/432733/ > > (I thought that is due to absense of SSE2 on first computer, but I've > > rechecked on Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with similar > > results). > > > > I'm getting that CPython is nearly 4-8 times faster than PyPy. > > Is it a bug in PyPy or what is wrong (may be "specific" to PyPy) in my > script? > > > > Alex. > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Fri Jul 8 03:05:18 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 08 Jul 2011 01:05:18 -0000 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: <20110708003048.GA8477@ubuntu> Message-ID: <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> On 12:38 am, alex.gaynor at gmail.com wrote: >repeat itself is not slow, it's just that when it's used it iterates >over >it, in RPython (meaning it's not jit'd) which results in a dictionary >lookup >for the next() method at every iteration, which is slowish, list hits a >special case so it doesn' thave that overhead. Is it time to reimplement repeat in Python then? Jean-Paul From alex.gaynor at gmail.com Fri Jul 8 03:15:08 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 7 Jul 2011 18:15:08 -0700 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> References: <20110708003048.GA8477@ubuntu> <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> Message-ID: No, you would need to implement list.__setitem__ in Python, which we could do, does the JIT see such code? Alex On Thu, Jul 7, 2011 at 6:05 PM, wrote: > On 12:38 am, alex.gaynor at gmail.com wrote: > >> repeat itself is not slow, it's just that when it's used it iterates over >> it, in RPython (meaning it's not jit'd) which results in a dictionary >> lookup >> for the next() method at every iteration, which is slowish, list hits a >> special case so it doesn' thave that overhead. >> > > Is it time to reimplement repeat in Python then? > > Jean-Paul > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Fri Jul 8 09:47:04 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 8 Jul 2011 09:47:04 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: Message-ID: On Fri, Jul 8, 2011 at 1:49 AM, Alex Gaynor wrote: > Hi all, > I've now spoken with several developers about the object identity that > arises out of hte new dict strategies, and all seem to think that the > current implementation breaks Python's semantics, that is: > x = 3 > z = {x: None} > assert next(iter(z)) is x > will fail. ?The only solution I see to this that maintains the correct > semantics in all cases is to specialize is/id() for primitive types. ?That > is to say, all integers of any given value have the same id() and x is y is > true. ?Doing x is y is true is easy, you simply have a multimethod which > dispatches on the types and compares intval for W_IntObjects (objects of > differing types can never have the same identity) however the question is > what to use for id() that is unique, never changes for an int, and doesn't > intersect with any other live object. ?In particular the last constraint is > the most difficult I think. > Alex >>> x = 3 >>> x is 3 True >>> x = 1003 >>> x is 1003 False >>> Stop relying on obscure details I think From holger at merlinux.eu Fri Jul 8 09:58:57 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 8 Jul 2011 07:58:57 +0000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: Message-ID: <20110708075857.GF20287@merlinux.eu> On Fri, Jul 08, 2011 at 09:47 +0200, Maciej Fijalkowski wrote: > On Fri, Jul 8, 2011 at 1:49 AM, Alex Gaynor wrote: > > Hi all, > > I've now spoken with several developers about the object identity that > > arises out of hte new dict strategies, and all seem to think that the > > current implementation breaks Python's semantics, that is: > > x = 3 > > z = {x: None} > > assert next(iter(z)) is x > > will fail. ?The only solution I see to this that maintains the correct > > semantics in all cases is to specialize is/id() for primitive types. ?That > > is to say, all integers of any given value have the same id() and x is y is > > true. ?Doing x is y is true is easy, you simply have a multimethod which > > dispatches on the types and compares intval for W_IntObjects (objects of > > differing types can never have the same identity) however the question is > > what to use for id() that is unique, never changes for an int, and doesn't > > intersect with any other live object. ?In particular the last constraint is > > the most difficult I think. > > Alex > > >>> x = 3 > >>> x is 3 > True > >>> x = 1003 > >>> x is 1003 > False > >>> I guess this is not what Alex means here. In CPython, PyPy (both 1.5 and trunk) you can do: >>> class A: pass >>> a=A() >>> next(iter({a: None})) is a True but not with ints on pypy-trunk >>> a=3 >>> next(iter({a: None})) is a False (it's true on pypy-1.5). IOW, i think the issue here is that iterating over keys of a dict usually gives the exact same ("is") objects in CPython whereas pypy trunk does not provide that at least for ints. best, holger > Stop relying on obscure details I think > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From william.leslie.ttg at gmail.com Fri Jul 8 10:31:37 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Fri, 8 Jul 2011 18:31:37 +1000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: <20110708075857.GF20287@merlinux.eu> References: <20110708075857.GF20287@merlinux.eu> Message-ID: On 8 July 2011 17:58, holger krekel wrote: > IOW, i think the issue here is that iterating over keys of a dict usually > gives the exact same ("is") objects in CPython whereas pypy trunk does not > provide that at least for ints. I couldn't find anything precise in the official documentation on the meaning of 'is'. I think that the general understanding is that it makes no sense whatsoever on immutable objects (as in, it isn't guaranteed to do so). Consequently, a python implementation could also cache tuples. Re-using tuples might sound unusual, but there are special cases that start to sound reasonable, such as caching the empty tuple, or copy-propogating a tuple unpack & repack. The language spec is very light on what is allowed to be a 'different object', and what is *suggested* by cpython's int caching behaviour is that the behaviour of 'is' for language-provided immutable objects can't be relied upon in any way, shape or form. Pypy hasn't matched cpython's behaviour with ints here in a long time, so it obviously doesn't matter. On another note: what Alex talks about as being two different cases are just one with the small int optimisation - all references can be compared by value in the C backend with small ints enabled, if the object space doesn't provide alternative behaviour. -- William Leslie From fijall at gmail.com Fri Jul 8 10:31:28 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 8 Jul 2011 10:31:28 +0200 Subject: [pypy-dev] pypy 1.5 on Windows: sqlite connection args In-Reply-To: References: Message-ID: On Sun, Jun 5, 2011 at 4:18 PM, Caleb Hattingh wrote: > In lib-pypy/_sqlite3.py, line 239 (in Win 1.5 stable release) in class > Connection(object): > > def __init__(self, database, isolation_level="", detect_types=0, > timeout=None, cached_statements=None, factory=None): > ...might need to become: > def __init__(self, database, isolation_level="", detect_types=0, > timeout=None, cached_statements=None, factory=None, > check_same_thread=False): Fixed on trunk From arigo at tunes.org Fri Jul 8 11:18:45 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Jul 2011 11:18:45 +0200 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: <20110708003048.GA8477@ubuntu> <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> Message-ID: Hi Alex, Before attacking the problem with the JIT, we should understand better why PyPy is 4-8 times slower than CPython. Normally you'd expect the factor to be at most 2. I suppose the answer is that our itertools.repeat() is bad for some reason. A bient?t, Armin. From arigo at tunes.org Fri Jul 8 11:52:10 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Jul 2011 11:52:10 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: Hi William, On Fri, Jul 8, 2011 at 10:31 AM, William ML Leslie wrote: > On another note: what Alex talks about as being two different cases > are just one with the small int optimisation - all references can be > compared by value in the C backend with small ints enabled, if the > object space doesn't provide alternative behaviour. No, because e.g. of longs. You don't have the same issue if you use longs instead of ints in this particular example, but more generally the issue exists too. A first note is that it's impossible to satisfy all three of Alex's criteria in general: we would like id(x) to be a unique word-sized number determined only by the value of 'x', and different for every value of 'x' and for every object of a different type too; but if the possible 'x'es are all possible word-sized integers, then it's impossible to satisfy this, just because there are too many of them. The problem only gets "more impossible" if we include all long objects in 'x'. The problem is not new, it is just a bit more apparent. For example, already in pypy 1.5 we have: >>>> a = A() >>>> d = a.__dict__ >>>> s = 'foobar' >>>> d[s] = 5 >>>> id(s) 163588812 >>>> id(d.keys()[0]) 163609508 >>>> id(d.keys()[0]) 163609520 I thought that there are also issues that would only show up with the JIT, because of the _immutable_ flag on W_IntObject, but it seems that I'm wrong. I can only say that Psyco has such issues, but nobody complained about them: lst = [] def f(x): for _ in range(3): pass # prevents inlining lst.append(x) def g(n): for i in range(n): f(i); f(i) With Psyco a call to g(5000) puts in 'lst' 10000 integer objects that are mostly all distinct objects, although there should in theory be at most 5000 distinct objects in there. (PyPy is safe so far because the call_assembler from g() to f() passes fully built W_IntObjects, instead of just cpu-level words; but that may change if in the future we add an optimization that knows how to generate a more efficient call_assembler.) I suppose that it's again a mixture of rules that are too vague and complains "but it works on CPython!" that are now being voiced just because the already-existing difference just became a bit more apparent. Sorry for the rant, I don't have an obvious solution :-) A bient?t, Armin. From cesare.di.mauro at gmail.com Fri Jul 8 12:04:18 2011 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Fri, 8 Jul 2011 12:04:18 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: 2011/7/8 Armin Rigo > Hi William, > > On Fri, Jul 8, 2011 at 10:31 AM, William ML Leslie > wrote: > > On another note: what Alex talks about as being two different cases > > are just one with the small int optimisation - all references can be > > compared by value in the C backend with small ints enabled, if the > > object space doesn't provide alternative behaviour. > > No, because e.g. of longs. You don't have the same issue if you use > longs instead of ints in this particular example, but more generally > the issue exists too. > > A first note is that it's impossible to satisfy all three of Alex's > criteria in general: we would like id(x) to be a unique word-sized > number determined only by the value of 'x', and different for every > value of 'x' and for every object of a different type too; but if the > possible 'x'es are all possible word-sized integers, then it's > impossible to satisfy this, just because there are too many of them. > The problem only gets "more impossible" if we include all long objects > in 'x'. > > The problem is not new, it is just a bit more apparent. For example, > already in pypy 1.5 we have: > > >>>> a = A() > >>>> d = a.__dict__ > >>>> s = 'foobar' > >>>> d[s] = 5 > >>>> id(s) > 163588812 > >>>> id(d.keys()[0]) > 163609508 > >>>> id(d.keys()[0]) > 163609520 > > I thought that there are also issues that would only show up with the > JIT, because of the _immutable_ flag on W_IntObject, but it seems that > I'm wrong. I can only say that Psyco has such issues, but nobody > complained about them: > > lst = [] > def f(x): > for _ in range(3): pass # prevents inlining > lst.append(x) > def g(n): > for i in range(n): > f(i); f(i) > > With Psyco a call to g(5000) puts in 'lst' 10000 integer objects that > are mostly all distinct objects, although there should in theory be at > most 5000 distinct objects in there. (PyPy is safe so far because the > call_assembler from g() to f() passes fully built W_IntObjects, > instead of just cpu-level words; but that may change if in the future > we add an optimization that knows how to generate a more efficient > call_assembler.) > > I suppose that it's again a mixture of rules that are too vague and > complains "but it works on CPython!" that are now being voiced just > because the already-existing difference just became a bit more > apparent. Sorry for the rant, I don't have an obvious solution :-) > > > A bient?t, > > Armin. > Hi Armin I fully agree. It's not an issue, but an implementation-specific detail which programmers don't have to assume always true. CPython can be compiled without "smallints" (-5..256, if I remember correctly) caching. There's a #DEFINE that can be disabled, so EVERY int (or long) will be allocated, so using the is operator will return False most of the time (unless you are just copied exactly the same object). The same applies for 1 character strings, which are USUALLY cached by CPython. So, there must be care about using is. It's safe for some trivial objects (None, False, True, Ellipsis) and, I think, with user-defined classes' instances, but not for everything. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From bokr at oz.net Fri Jul 8 16:07:53 2011 From: bokr at oz.net (Bengt Richter) Date: Fri, 08 Jul 2011 16:07:53 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: On 07/08/2011 10:31 AM William ML Leslie wrote: > On 8 July 2011 17:58, holger krekel wrote: >> IOW, i think the issue here is that iterating over keys of a dict usually >> gives the exact same ("is") objects in CPython whereas pypy trunk does not >> provide that at least for ints. > > I couldn't find anything precise in the official documentation on the > meaning of 'is'. I think that the general understanding is that it > makes no sense whatsoever on immutable objects (as in, it isn't > guaranteed to do so). > > Consequently, a python implementation could also cache tuples. > Re-using tuples might sound unusual, but there are special cases that > start to sound reasonable, such as caching the empty tuple, or > copy-propogating a tuple unpack& repack. The language spec is very > light on what is allowed to be a 'different object', and what is > *suggested* by cpython's int caching behaviour is that the behaviour > of 'is' for language-provided immutable objects can't be relied upon > in any way, shape or form. > > Pypy hasn't matched cpython's behaviour with ints here in a long time, > so it obviously doesn't matter. > > On another note: what Alex talks about as being two different cases > are just one with the small int optimisation - all references can be > compared by value in the C backend with small ints enabled, if the > object space doesn't provide alternative behaviour. > python -c 'import this'|egrep 'rules|purity|hard' #;-) From arigo at tunes.org Fri Jul 8 16:07:07 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Jul 2011 16:07:07 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: Hi all, On Fri, Jul 8, 2011 at 12:04 PM, Cesare Di Mauro wrote: > So, there must be care about using is. It's safe for some trivial objects > (None, False, True, Ellipsis) and, I think, with user-defined classes' > instances, but not for everything. The problem is more acute with id(). Some standard library modules like copy.py *need* a working id() for any object, including immutable ones, because CPython has no identity dict. After some discussion with Carl Friedrich, here is the best we could think of. Say that "a is b" is redefined as being True for two equal immutable objects of the same type. Then we want the following property to always hold: "a is b" if and only if "id(a) == id(b)". We can do that by having id(x) return either a regular integer or a long. Let's call for the rest of this discussion an "immutable object" an object of type int, long or float. If 'x' is not an immutable object, then id(x) can return the same value as it does now. If 'x' is an immutable object, then we compute from it a long value that does *not* fit in 32- or 64-bit, and that includes some tagging to make sure that immutable objects of different types have different id's. For example, id(7) would be (2**32 + 7<<3 + 1). Such a solution should make two common use cases of id() work: 1. as keys in a dictionary, to implement an identity dict, like in copy.py: in this case we take the id() of random objects including immutable ones, but only expect the result to work as keys in the dictionary. Getting arbitrarily-sized longs is not a problem here. 2. more contrived examples involve taking the id() of some instance, sending it around as a (word-sized) integer, and when getting it back retrieving the original instance from some dict. I don't expect people to do that with immutable objects, only their own custom instances. That's why it should be enough if id(x) returns a regular, 32- or 64-bit integer for *non-immutable* objects. A bient?t, Armin. From amauryfa at gmail.com Fri Jul 8 16:14:37 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 8 Jul 2011 16:14:37 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: 2011/7/8 Cesare Di Mauro : > I fully agree. It's not an issue, but an implementation-specific detail > which programmers don't have to assume always true. > > CPython can be compiled without "smallints" (-5..256, if I remember > correctly) caching. There's a #DEFINE that can be disabled, so EVERY int (or > long) will be allocated, so using the is operator will return False most of > the time (unless you are just copied exactly the same object). > > The same applies for 1 character strings, which are USUALLY cached by > CPython. But the problem here is not object cache, but preservation of object identity, which is quite different. Python containers are supposed to keep the objects you put inside: myList.append(x) assert myList(-1) is x myDict[x] = 1 for key in myDict: if key is x: ... -- Amaury Forgeot d'Arc From fijall at gmail.com Fri Jul 8 16:17:08 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 8 Jul 2011 16:17:08 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: On Fri, Jul 8, 2011 at 4:14 PM, Amaury Forgeot d'Arc wrote: > 2011/7/8 Cesare Di Mauro : >> I fully agree. It's not an issue, but an implementation-specific detail >> which programmers don't have to assume always true. >> >> CPython can be compiled without "smallints" (-5..256, if I remember >> correctly) caching. There's a #DEFINE that can be disabled, so EVERY int (or >> long) will be allocated, so using the is operator will return False most of >> the time (unless you are just copied exactly the same object). >> >> The same applies for 1 character strings, which are USUALLY cached by >> CPython. > > But the problem here is not object cache, but preservation of object identity, > which is quite different. > Python containers are supposed to keep the objects you put inside: [citation needed] array.array does not for one > > myList.append(x) > assert myList(-1) is x > > myDict[x] = 1 > for key in myDict: > ? ?if key is x: > ? ? ? ?... > also dict doesn't work if you overwrite the key: d = {1003: None} x = 1003 d[x] = None d.keys()[0] is x From cfbolz at gmx.de Fri Jul 8 16:50:05 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 08 Jul 2011 16:50:05 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: <4E17191D.6010700@gmx.de> On 07/08/2011 04:14 PM, Amaury Forgeot d'Arc wrote: > 2011/7/8 Cesare Di Mauro: >> I fully agree. It's not an issue, but an implementation-specific detail >> which programmers don't have to assume always true. >> >> CPython can be compiled without "smallints" (-5..256, if I remember >> correctly) caching. There's a #DEFINE that can be disabled, so EVERY int (or >> long) will be allocated, so using the is operator will return False most of >> the time (unless you are just copied exactly the same object). >> >> The same applies for 1 character strings, which are USUALLY cached by >> CPython. > > But the problem here is not object cache, but preservation of object identity, > which is quite different. I think in the end id is the hard problem. Object identity can be fixed. We could say that "a is b" uses equality for primitive objects. However, then id needs to be fixed too, so that the following property holds: "a is b" if and only if "id(a) == id(b)" This is the harder part. Carl Friedrich From anto.cuni at gmail.com Fri Jul 8 17:10:58 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 08 Jul 2011 17:10:58 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: <4E17191D.6010700@gmx.de> References: <20110708075857.GF20287@merlinux.eu> <4E17191D.6010700@gmx.de> Message-ID: <4E171E02.40202@gmail.com> On 08/07/11 16:50, Carl Friedrich Bolz wrote: > > I think in the end id is the hard problem. Object identity can be fixed. We > could say that "a is b" uses equality for primitive objects. what about starting to think about solutions only when we are sure that it's actually a problem? I don't expect any real world code to rely on this behavior (or, if it does, I'm up to call it broken :-)) From arigo at tunes.org Fri Jul 8 17:19:17 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Jul 2011 17:19:17 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: <4E171E02.40202@gmail.com> References: <20110708075857.GF20287@merlinux.eu> <4E17191D.6010700@gmx.de> <4E171E02.40202@gmail.com> Message-ID: Hi Anto, On Fri, Jul 8, 2011 at 5:10 PM, Antonio Cuni wrote: > what about starting to think about solutions only when we are sure that it's > actually a problem? > > I don't expect any real world code to rely on this behavior (or, if it does, > I'm up to call it broken :-)) As I said in my previous e-mail, I think that e.g. copy.py relies on such behavior, and more generally any Python code that has to use id() to emulate an identity-dict --- as broken as that approach is, just because CPython thought that identity-dicts were unnecessary. It may not actually be a problem right now in copy.py, but still, I fear that it's a problem waiting to hurt us. A bient?t, Armin. From anto.cuni at gmail.com Fri Jul 8 17:40:21 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 08 Jul 2011 17:40:21 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E17191D.6010700@gmx.de> <4E171E02.40202@gmail.com> Message-ID: <4E1724E5.2030902@gmail.com> On 08/07/11 17:19, Armin Rigo wrote: > As I said in my previous e-mail, I think that e.g. copy.py relies on > such behavior, and more generally any Python code that has to use id() > to emulate an identity-dict --- as broken as that approach is, just > because CPython thought that identity-dicts were unnecessary. > > It may not actually be a problem right now in copy.py, but still, I > fear that it's a problem waiting to hurt us. I don't think that copy.py relies on this behavior (even after looking at the code, but I might wrong). My point is that we are not breaking id(); identity dicts emulated using id() continue to work as normal. The only assumption that we are breaking is this one, which has nothing to do with identity dicts or id(). d = {} d[x] = None assert d.keys()[0] is x As fijal points out, the semantics of this particular behavior is already half-broken, even in CPython. E.g., in the following example we never remove anything from the dictionary, nevertheless a key "disappears": x = 1003 d = {x: None} assert d.keys()[0] is x d[1000+3] = None assert d.keys()[0] is x # BOOM ciao, Anto From exarkun at twistedmatrix.com Fri Jul 8 17:04:50 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 08 Jul 2011 15:04:50 -0000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: <20110708150450.3761.405889280.divmod.xquotient.121@localhost.localdomain> On 02:17 pm, fijall at gmail.com wrote: >On Fri, Jul 8, 2011 at 4:14 PM, Amaury Forgeot d'Arc > wrote: >>2011/7/8 Cesare Di Mauro : >>>I fully agree. It's not an issue, but an implementation-specific >>>detail >>>which programmers don't have to assume always true. >>> >>>CPython can be compiled without "smallints" (-5..256, if I remember >>>correctly) caching. There's a #DEFINE that can be disabled, so EVERY >>>int (or >>>long) will be allocated, so using the is operator will return False >>>most of >>>the time (unless you are just copied exactly the same object). >>> >>>The same applies for 1 character strings, which are USUALLY cached by >>>CPython. >> >>But the problem here is not object cache, but preservation of object >>identity, >>which is quite different. >>Python containers are supposed to keep the objects you put inside: > >[citation needed] array.array does not for one Yes, and array.array is weird. :) It either exists as a memory optimization (ie, I don't want objects) or a way to directly lay out memory (to pass to a C API). Either way, you can't put arbitrary objects into it either - so it's already a little special, even if you disregard the fact that it doesn't preserve the identify the objects you can put into it. However, you're right. It exists, and it has this non-identity- preserving behavior. Is it a good thing, though? Or just an accident of how someone tried to let CPython be faster for some types of problems? >> >>myList.append(x) >>assert myList(-1) is x >> >>myDict[x] = 1 >>for key in myDict: >>? ?if key is x: >>? ? ? ?... > >also dict doesn't work if you overwrite the key: > >d = {1003: None} >x = 1003 >d[x] = None >d.keys()[0] is x This doesn't invalidate the original point, as far as I can tell. It just demonstrates again that you can have two instances of 1003. Whether dict guarantees to always use the new key or the old key when an update is made is a separate question. I think it would be better if object identity didn't depend on this mysterious quality of "immutability". The language is easier to understand (particularly for new programmers) if one can talk about objects and references without having to also explain that _some_ data types are represented using things that are sort of like objects but not quite (and worse if it depends on what types the JIT feels like playing with in any particular version of the interpreter). Jean-Paul From arigo at tunes.org Fri Jul 8 17:59:03 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Jul 2011 17:59:03 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: <4E1724E5.2030902@gmail.com> References: <20110708075857.GF20287@merlinux.eu> <4E17191D.6010700@gmx.de> <4E171E02.40202@gmail.com> <4E1724E5.2030902@gmail.com> Message-ID: Hi Anto, On Fri, Jul 8, 2011 at 5:40 PM, Antonio Cuni wrote: > My point is that we are not breaking id(); identity dicts emulated using id() > continue to work as normal. Well, the argument goes like this: *if* we think that the problems like the one you describe are to be fixed, then we really need to hack at "is", and *then* we have a problem with id() as well, and we have to fix it as I described. You can also try to argue that nothing is broken so far, neither in "is" nor in id() nor your examples. I fear that we are going to end up seeing more and more cases where users rely on the current CPython behavior, particularly because we're going to expose such issues more and more over time as we add new optimizations. But I may be wrong and it may be enough to document it in cpython-differences.rst. > x = 1003 > d = {x: None} > assert d.keys()[0] is x > d[1000+3] = None > assert d.keys()[0] is x # BOOM This is the wrong example: in this case, CPython guarantees that it will not crash. The semantics are not really half-broken, just not written out clearly: when you add an object to a dict, if there is another key already in there that compares equal, then the existing key is kept; it is not replaced by the new key. (Both CPython and PyPy agree to this rule.) A bient?t, Armin. From anto.cuni at gmail.com Fri Jul 8 18:37:55 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 08 Jul 2011 18:37:55 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E17191D.6010700@gmx.de> <4E171E02.40202@gmail.com> <4E1724E5.2030902@gmail.com> Message-ID: <4E173263.2060603@gmail.com> On 08/07/11 17:59, Armin Rigo wrote: > I fear that we are going to end up seeing more and more cases where > users rely on the current CPython behavior, particularly because we're > going to expose such issues more and more over time as we add new > optimizations. But I may be wrong and it may be enough to document it > in cpython-differences.rst. yes, the whole point of my position is that I don't think there is many code around relying on this behavior. Thus, I propose to wait a bit and see how many people complain, before fixing what it might be a non-issue >> x = 1003 >> d = {x: None} >> assert d.keys()[0] is x >> d[1000+3] = None >> assert d.keys()[0] is x # BOOM > > This is the wrong example: in this case, CPython guarantees that it > will not crash. The semantics are not really half-broken, just not > written out clearly: when you add an object to a dict, if there is > another key already in there that compares equal, then the existing > key is kept; it is not replaced by the new key. (Both CPython and > PyPy agree to this rule.) ouch, I should write test for my emails :-) From arigo at tunes.org Fri Jul 8 23:40:04 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 8 Jul 2011 23:40:04 +0200 Subject: [pypy-dev] Floating point computation In-Reply-To: References: Message-ID: Hi, On Thu, Jul 7, 2011 at 10:59 AM, Maciej Fijalkowski wrote: >> The Translation and JIT backend for PyPy may be able to allow Python >> programmers to use SIMD instructions directly from Python. > > We kind of want to do that automatically for numpy operations. Can all basic SIMD instructions be mapped to numpy array operations? If so, why doesn't numpy already use them? (Or does it?) And if not, then is there some existing C extension module for CPython that exposes them somehow? Of course CPython cannot hope for impressive speedups with such an extension module, but such a module in PyPy could be JITted and give massive speed benefits. A bient?t, Armin. From william.leslie.ttg at gmail.com Sat Jul 9 05:20:53 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Sat, 9 Jul 2011 13:20:53 +1000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: On 8 July 2011 19:52, Armin Rigo wrote: > Hi William, Hi Armin, everybody, > On Fri, Jul 8, 2011 at 10:31 AM, William ML Leslie > wrote: >> On another note: what Alex talks about as being two different cases >> are just one with the small int optimisation - all references can be >> compared by value in the C backend with small ints enabled, if the >> object space doesn't provide alternative behaviour. > > No, because e.g. of longs. You don't have the same issue if you use > longs instead of ints in this particular example, but more generally > the issue exists too. > > A first note is that it's impossible to satisfy all three of Alex's > criteria in general: we would like id(x) to be a unique word-sized > number determined only by the value of 'x', and different for every > value of 'x' and for every object of a different type too; but if the > possible 'x'es are all possible word-sized integers, then it's > impossible to satisfy this, just because there are too many of them. > The problem only gets "more impossible" if we include all long objects > in 'x'. That id(x) be a word-sized value uniquely determined by the value of x is impossible, yes, as the following program should demonstrate: while True: id([]) We create an infinite number of objects here, and if each one had to have a unique word-sized id, we'd exhaust that space pretty quickly. What id() does provide is that as long as there is a *reference* to the object, it returns a constant and unique integer (which we are assuming should be a word-sized integer). As later emails on this list suggested, it would be better if the semantics were relaxed to "the id should be preserved", meaning that placement of an integer or string into a container should allow it to have the same id on retrieval. New objects resulting from operations such as multiplication or string concatenation need not have the same id as other objects with the same value - if we did that for strings, it would have serious consequences for parallel code. What I was suggesting is that, since every live object must be encoded in an object reference somewhere, that object references should be at least good enough to suggest an id. My point about small integers wasn't really about word-sized ints, I was talking about the smallint optimisation, which as I understood it, boxes small app-level integers in the C backend. It does this by shifting integers right and bitwise-oring with 1; encoding the integer into the reference. "By definition", as Carl put it, there are never more objects represented in this way than we can fit in a reasonable, bounded id size. The suggestion then is to use the value of the object reference cast as a word-sized integer as the id of the object for integers so encoded, and calculate the id of other objects in the usual fashion. This happens to work for larger integers and floats, because the id will be preserved as long as a reference to them exists by their boxedness. Floats could use a similar mechanism to integers, eg, their bit representation shifted left two and bitwise-ored with 2. That does mean that id() is no longer word-sized, but it does not make it unbounded. > The problem is not new, it is just a bit more apparent. For example, > already in pypy 1.5 we have: > >>>>> a = A() >>>>> d = a.__dict__ >>>>> s = 'foobar' >>>>> d[s] = 5 >>>>> id(s) > 163588812 >>>>> id(d.keys()[0]) > 163609508 >>>>> id(d.keys()[0]) > 163609520 What is keeping us from using the underlying rpython string as the basis for id? I guess there is nowhere obvious to store the id or the need to store the id in the gc copy phase. In that way, it'd make for an ugly special case. On 9 July 2011 00:07, Armin Rigo wrote: > After some discussion with Carl Friedrich, here is the best we could > think of. Say that "a is b" is redefined as being True for two equal > immutable objects of the same type. Then we want the following > property to always hold: "a is b" if and only if "id(a) == id(b)". I would prefer to weaken this just a little: x is y iff id(x) == id(y) WHEN x and y are live for the duration of the equality. This counters cases such as: id([]) == id([]) which can certainly happen under any reasonable definition of id. > We > can do that by having id(x) return either a regular integer or a long. > Let's call for the rest of this discussion an "immutable object" an > object of type int, long or float. If 'x' is not an immutable object, > then id(x) can return the same value as it does now. If 'x' is an > immutable object, then we compute from it a long value that does *not* > fit in 32- or 64-bit, and that includes some tagging to make sure that > immutable objects of different types have different id's. For > example, id(7) would be (2**32 + 7<<3 + 1). This makes the range of id() unbounded, where its domain is bounded. I don't feel comfortable with it, although I know that isn't much of an objection. > Such a solution should make two common use cases of id() work: > > 1. as keys in a dictionary, to implement an identity dict, like in > copy.py: in this case we take the id() of random objects including > immutable ones, but only expect the result to work as keys in the > dictionary. Getting arbitrarily-sized longs is not a problem here. Speaking of, maybe it'd be easier to attempt to get the identity dict into the language proper. William Leslie From stefan_ml at behnel.de Sat Jul 9 06:26:46 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 09 Jul 2011 06:26:46 +0200 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: <20110708003048.GA8477@ubuntu> <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> Message-ID: Armin Rigo, 08.07.2011 11:18: > Before attacking the problem with the JIT, we should understand better > why PyPy is 4-8 times slower than CPython. Normally you'd expect the > factor to be at most 2. I suppose the answer is that our > itertools.repeat() is bad for some reason. You shouldn't forget that itertools contains extremely well optimised C code. I recently tried to (algorithmically) optimise the pure Python implementations from the CPython docs for Cython, and ended up with timings between 10% faster and 50% slower. It's actually hard to reach up to the raw power of hand tuned C code here. http://blog.behnel.de/index.php?p=163 http://blog.behnel.de/index.php?p=185 Given that some of the functions don't execute anything more than a couple of assembly instructions per cycle, you'll have to send a really well tuned high level implementation into the race to even come close. I would expect that PyPy's implementation simply wasn't written for that. That being said, it would be nice to have an itertools micro benchmark for the common Python test suite. Stefan From alex.gaynor at gmail.com Sat Jul 9 06:40:18 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 8 Jul 2011 21:40:18 -0700 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: <20110708003048.GA8477@ubuntu> <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> Message-ID: I'm not too sure what could be better wrong with it, it's rather short: https://bitbucket.org/pypy/pypy/src/default/pypy/module/itertools/interp_itertools.py#cl-85 Alex On Fri, Jul 8, 2011 at 2:18 AM, Armin Rigo wrote: > Hi Alex, > > Before attacking the problem with the JIT, we should understand better > why PyPy is 4-8 times slower than CPython. Normally you'd expect the > factor to be at most 2. I suppose the answer is that our > itertools.repeat() is bad for some reason. > > > A bient?t, > > Armin. > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sat Jul 9 10:17:33 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 9 Jul 2011 10:17:33 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: Hi, On Sat, Jul 9, 2011 at 5:20 AM, William ML Leslie wrote: > My point about small integers (...) I think that your point about small integers is broken (even assuming that smallints are enabled by default, which is not the case). It means that we'd get an id() function which "behaves" as long as taking the id() of a 31-bit signed integer, and then doesn't behave as expected neither for full 32-bit integers, nor for longs, nor for floats. > This happens to work for larger integers and floats, because the id > will be preserved as long as a reference to them exists by their > boxedness. > Floats could use a similar mechanism to integers, eg, their bit > representation shifted left two and bitwise-ored with 2. I don't understand these two sentences, because they seem to say the exact opposite of each other... >?That does mean that id() is no longer word-sized, but it does not make it > unbounded. The "unbounded" part in my e-mail was about longs. Obviously if you are computing id(x) where x is in some finite set (like ints or floats), then the results are in some finite set too. > What is keeping us from using the underlying rpython string as the > basis for id? This is probably a good enough workaround for strings and unicodes. > Speaking of, maybe it'd be easier to attempt to get the identity dict > into the language proper. We tried at some point, but python-dev refused the idea. Maybe the idea has more chances for approval now that we can really show with performance numbers that it's a deep issue, as opposed to just wave our hands. Feel free to try again. In the meantime I've decided, at least myself, to stick with the approach that Python is whatever is in 2.7, and that we have to work around such issues instead of fixing them properly in the language. A bient?t, Armin. From osadchiy.ilya at gmail.com Sat Jul 9 11:19:26 2011 From: osadchiy.ilya at gmail.com (Ilya Osadchiy) Date: Sat, 9 Jul 2011 12:19:26 +0300 Subject: [pypy-dev] Floating point computation Message-ID: >>> The Translation and JIT backend for PyPy may be able to allow Python >>> programmers to use SIMD instructions directly from Python. >> >> We kind of want to do that automatically for numpy operations. > > Can all basic SIMD instructions be mapped to numpy array operations? > If so, why doesn't numpy already use them? ?(Or does it?) > I think CPython's numpy may be compiled with auto-vectorization and therefore use SIMD. It also has optional dependency for lapack, and AFAIK most lapacks use SIMD. > > And if not, then is there some existing C extension module for CPython > that exposes them somehow? ?Of course CPython cannot hope for > impressive speedups with such an extension module, but such a module > in PyPy could be JITted and give massive speed benefits. > I think that JITting element-wise operations to SIMD assembly is feasible. Logically you can see it like: normal JIT => loop unroll => instruction reordering => substitute N instructions of same type with 1 simd instruction Of course there are problems (loads and stores need special treatment, not every instruction has simd equivalent, register allocation differs, etc) But I think the bigger problem is JITting SIMD for things like matrix multiply and fft, which are more than a single loop and the order of calculations may seriously differ between scalar and simd versions. From neppord at gmail.com Sat Jul 9 19:24:08 2011 From: neppord at gmail.com (Samuel Ytterbrink) Date: Sat, 9 Jul 2011 19:24:08 +0200 Subject: [pypy-dev] integrity of code Message-ID: Hi! I was just thinking and then ask my self... Dose writing things in RPython and then compile the code with translate and a c compiler make it as hard to read as a c program... thinking that python some times get a hard time being a commercial tool for app making, companies don't want end users to be able to read there code. Am i correct and if so are there any improvements you could do to make the interpreter read a more 'seacret' version of python code, some kinde of binary code( or dose the byetcode work for this). Hope this is not a stupid or foolish topic. -- //Samuel Ytterbrink -------------- next part -------------- An HTML attachment was scrubbed... URL: From ademan555 at gmail.com Sun Jul 10 00:14:08 2011 From: ademan555 at gmail.com (Dan Roberts) Date: Sat, 9 Jul 2011 15:14:08 -0700 Subject: [pypy-dev] integrity of code In-Reply-To: References: Message-ID: Hi Samuel, If I understand your question correctly, you are a bit confused about RPython. Only the interpreter is translated to c and then compiled. The source code to the interpreter is obviously not secret at all, we all work on it and share it on bitbucket. User code might be secret, but it is not passed through the translation toolchain. In the case of Python on PyPy the user code is compiled to a bytecode very similar to CPython's bytecode. I would say this bytecode is quite readable, so it's significantly more readable than optimized x86 output. I suppose it would be possible to add some obfuscation to the bytecode but I don't know how one would do that effectively without modifying the entire bytecode interpreter and compiler, which would be an undertaking... I may have misunderstood you though, feel free to correct me. Cheers, Dan On Jul 9, 2011 10:24 AM, "Samuel Ytterbrink" wrote: > Hi! > > I was just thinking and then ask my self... Dose writing things in RPython > and then compile the code with translate and a c compiler make it as hard to > read as a c program... thinking that python some times get a hard time being > a commercial tool for app making, companies don't want end users to be able > to read there code. > > Am i correct and if so are there any improvements you could do to make the > interpreter read a more 'seacret' version of python code, some kinde of > binary code( or dose the byetcode work for this). > > Hope this is not a stupid or foolish topic. > > -- > //Samuel Ytterbrink -------------- next part -------------- An HTML attachment was scrubbed... URL: From bokr at oz.net Sun Jul 10 15:48:08 2011 From: bokr at oz.net (Bengt Richter) Date: Sun, 10 Jul 2011 15:48:08 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: On 07/09/2011 10:17 AM Armin Rigo wrote: > Hi, > > On Sat, Jul 9, 2011 at 5:20 AM, William ML Leslie > wrote: >> My point about small integers (...) > > I think that your point about small integers is broken (even assuming > that smallints are enabled by default, which is not the case). It > means that we'd get an id() function which "behaves" as long as taking > the id() of a 31-bit signed integer, and then doesn't behave as > expected neither for full 32-bit integers, nor for longs, nor for > floats. > >> This happens to work for larger integers and floats, because the id >> will be preserved as long as a reference to them exists by their >> boxedness. >> Floats could use a similar mechanism to integers, eg, their bit >> representation shifted left two and bitwise-ored with 2. > > I don't understand these two sentences, because they seem to say the > exact opposite of each other... > >> That does mean that id() is no longer word-sized, but it does not make it >> unbounded. > > The "unbounded" part in my e-mail was about longs. Obviously if you > are computing id(x) where x is in some finite set (like ints or > floats), then the results are in some finite set too. > >> What is keeping us from using the underlying rpython string as the >> basis for id? > > This is probably a good enough workaround for strings and unicodes. > >> Speaking of, maybe it'd be easier to attempt to get the identity dict >> into the language proper. > > We tried at some point, but python-dev refused the idea. Maybe the > idea has more chances for approval now that we can really show with > performance numbers that it's a deep issue, as opposed to just wave > our hands. Feel free to try again. In the meantime I've decided, at > least myself, to stick with the approach that Python is whatever is in > 2.7, and that we have to work around such issues instead of fixing > them properly in the language. > Does that mean you have to follow the vagaries of the 2.7 compiler optimizations (or lack of them, as the case may be) that make for results like these?: (note fresh 2.7.2 build below ;-) [& BTW kudos to those who made it easy with the tarball and config, make] [10:30 ~]$ python Python 2.7.2 (default, Jul 8 2011, 23:38:53) [GCC 4.1.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from ut.miscutil import disev,disex >>> id(2000) == id(2000) True >>> id(2000) == id(20*100) False >>> disev('id(2000) == id(2000)') 1 0 LOAD_NAME 0 (id) 3 LOAD_CONST 0 (2000) 6 CALL_FUNCTION 1 9 LOAD_NAME 0 (id) 12 LOAD_CONST 0 (2000) 15 CALL_FUNCTION 1 18 COMPARE_OP 2 (==) 21 RETURN_VALUE >>> disev('id(2000) == id(20*100)') 1 0 LOAD_NAME 0 (id) 3 LOAD_CONST 0 (2000) 6 CALL_FUNCTION 1 9 LOAD_NAME 0 (id) 12 LOAD_CONST 3 (2000) 15 CALL_FUNCTION 1 18 COMPARE_OP 2 (==) 21 RETURN_VALUE >>> Notice that 20*100 was folded to 2000, but a different constant was generated, hence, presumably, the different id. Will the next update of 2.7 do the same? BTW, disev above is just from a grabbag module of mine, amounting to def disev(s): import dis return dis.dis(compile(s,'disev','eval')) It doesn't have to be about integers: >>> id([]) == id([]) True >>> id(list()) == id(list()) False >>> disev('id([]) == id([])') 1 0 LOAD_NAME 0 (id) 3 BUILD_LIST 0 6 CALL_FUNCTION 1 9 LOAD_NAME 0 (id) 12 BUILD_LIST 0 15 CALL_FUNCTION 1 18 COMPARE_OP 2 (==) 21 RETURN_VALUE >>> disev('id(list()) == id(list())') 1 0 LOAD_NAME 0 (id) 3 LOAD_NAME 1 (list) 6 CALL_FUNCTION 0 9 CALL_FUNCTION 1 12 LOAD_NAME 0 (id) 15 LOAD_NAME 1 (list) 18 CALL_FUNCTION 0 21 CALL_FUNCTION 1 24 COMPARE_OP 2 (==) 27 RETURN_VALUE >>> Of course, the is operator call will keep both references alive on the stack, so whereas you get >>> id([]) == id([]) True >>> id([]), id([]) (3082823052L, 3082823052L) Keeping the arguments alive simultaneously gives >>> (lambda x,y:(id(x),id(y)))([],[]) (3082823052L, 3082881516L) >>> (lambda x,y:(id(x)==id(y)))([],[]) False >>> ... assuming the lambda above approximates what `is' does. Using id comparisons instead of `is' can look superficially a bit weird: >>> id([1]) == id([2]) True >>> Bottom line: id does not seem to have a complete abstract definition[1][2], so how may one define a 'correct' implementation? ISTM id now (CPython 2.7.2) is more like a version-dependent debugger function that can be useful in app hacks, but with caveats ;-) Regards, Bengt Richter [1] http://docs.python.org/reference/datamodel.html#index-824 [2] http://docs.python.org/library/functions.html#id PS. How does the following cachingid.py compare to your idea of what id should ideally do? (not tested except example) (Obviously it is just a concept implementation, not resource efficient) ________________________________________ refval = id # for using old id def id(x, objcache = {}): if type(x) in (int,float,bool,tuple,str,unicode): # etc t = (type(x), x) return refval(objcache.setdefault(t, t)[1]) else: return refval(x) # as now? XXX what is abstract meaning of id(some_mutable)? ________________________________________ It tries to return the ordinary id of the first instance of equivalent immutable objects passed to it, and otherwise uses the old id, which I'm not finished thinking about ;-) At least it fixes the difference between id(2000) and id(20*100): >>> oldid=id >>> from ut.cachingid import id # ut is just my utility collection >>> oldid(2000)==oldid(2000) True >>> oldid(2000)==oldid(20*100) False >>> id(2000)==id(2000) True >>> id(2000)==id(20*100) True >>> id.func_defaults ({(, 2000): (, 2000)},) >>> > > A bient?t, > > Armin. From arigo at tunes.org Sun Jul 10 16:09:11 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 10 Jul 2011 16:09:11 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: Hi Bengt, On Sun, Jul 10, 2011 at 3:48 PM, Bengt Richter wrote: > ?>>> id([1]) == id([2]) > True As pointed out by Carl Friedrich, the real definition of "id" is: * if x and y are two variables, then "x is y" <=> "id(x) == id(y)". That's why in any Python implementation, >>> x=[1]; y=[2]; id(x) == id(y) must return False, but not necessarily id([1]) == id([2]). A bient?t, Armin. From bokr at oz.net Sun Jul 10 18:08:41 2011 From: bokr at oz.net (Bengt Richter) Date: Sun, 10 Jul 2011 18:08:41 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: <4E19CE89.6070905@oz.net> On 07/10/2011 04:09 PM Armin Rigo wrote: > Hi Bengt, > > On Sun, Jul 10, 2011 at 3:48 PM, Bengt Richter wrote: >> >>> id([1]) == id([2]) >> True > True, I did write that. The key word in my line before that was `superficially' ;-) > As pointed out by Carl Friedrich, the real definition of "id" is: > > * if x and y are two variables, then "x is y"<=> "id(x) == id(y)". > So conceptually, id is about "variables" -- not objects? Isn't a "variable" just a special case of an expression (yielding a reference to a result object, like other expressions)? ISTM that they key thing above is that "if x and y are two variables" the two (or one if same) referenced objects are guaranteed to be live at the same time. So they can't be in the same "location" unless they are the same object. Someone else mentioned this liveness issue too. > That's why in any Python implementation, > >>>> x=[1]; y=[2]; id(x) == id(y) > > must return False, but not necessarily id([1]) == id([2]). > The definition with "two variables" seems unnecessarily restrictive, unless I am wrong that x=[1]; id(x) == id([2]) must also return false, since the x reference guarantees that the [2] cannot be created in the same "location" as the [1] held by the x. Something about returning location from id reminds me of the need for automatic closure creation in another context. Letting the expression result die and returning a kind of pointer to where the result object *was* seems like a dangling pointer problem, except I guess you can't dereference an id value (without hackery). Maybe id should raise an exception if the argument referenced only has a ref count of 1 (i.e., just the reference from the argument list)? Or else let id be a class and return a minimal instance only binding the passed object, and customize the compare ops to take into account type diffs etc.? Then there would be no id values without corresponding objects, and id values used in expressions would die a natural death, along with their references to their objects -- whether "variables" or expressions. Sorry to belabor the obvious ;-) Regards, Bengt > > A bient?t, > > Armin. From lac at openend.se Sun Jul 10 21:13:26 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 10 Jul 2011 21:13:26 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: Message from Bengt Richter of "Sun, 10 Jul 2011 18:08:41 +0200." <4E19CE89.6070905@oz.net> References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> Message-ID: <201107101913.p6AJDQm6027255@theraft.openend.se> What do we want to happen when somebody -- say in a C extension -- takes the id of an object that is scheduled to be removed when the gc next runs? Laura From bokr at oz.net Mon Jul 11 12:29:34 2011 From: bokr at oz.net (Bengt Richter) Date: Mon, 11 Jul 2011 12:29:34 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: <201107101913.p6AJDQm6027255@theraft.openend.se> References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> <201107101913.p6AJDQm6027255@theraft.openend.se> Message-ID: On 07/10/2011 09:13 PM Laura Creighton wrote: > What do we want to happen when somebody -- say in a C extension -- takes the id of an object > that is scheduled to be removed when the gc next runs? IMO taking the id should increment the object ref counter and prevent the garbage collection, until the id value itself is garbage collected. The obvious way would be to make id an object that keeps a reference to the object whose id it represents. See below[1] for an example (just for discussion illustration). Of course in low level access, conventions of ownership can sometimes safely optimize away actual ref incr/decr, but it sounds like your example proposes "taking an id" after ref count has gone to zero. That's like doing a reinterpret_cast to integer of a malloc pointer after the area's been freed, and expecting it to mean something. It should be an enforced nono, if you ask me. The example id attempts to make all equivalent immutables of the same type have the same id, by taking advantage of dict's key comparison properties and the .setdefault method. to get something like the current, with just the object reference to make an id hold its object, use IdHolder in place of Id. But then you can get all kinds of ids for the same immutable value, as with the old id. _______________________________________________________________________________ # idstuff.py -- a concept-exploring toy re "id" # 2011-07-10 22:58:46 bokr # class Id(object): refval = id # old id function objcache = {} # to store typed immutables for ref to first of equal-valued encountered # and using old id of the first as id for all of same type and equal value. def __init__(self, obj): self.obj = obj def id(self): so=self.obj if type(so) in (int,float,bool,tuple,str,unicode): # etc? t = (type(so), so) return self.refval(self.objcache.setdefault(t, t)[1]) else: return self.refval(so) # as now? XXX what is abstract meaning of id(some_mutable)? def __eq__(self, other): if type(self) != type(other): raise TypeError('Id instances can only be compared with each other,' 'not to "%s" instances.'% type(other).__name__) tobj=type(self.obj) tother=type(other.obj) if tobj != tother: return False return self.id() == other.id() def __repr__(self): return ''%(self.obj, self.id(), self.refval(self.obj)) def __str__(self): return ''%(self.obj,) class IdHolder(object): refval = id # old id function def __init__(self, obj): self.obj = obj self.id = self.refval(obj) def __eq__(self, other): if type(self) != type(other): raise TypeError('IdHolder instances can only be compared with each other,' ' not to "%s" instances.'% type(other).__name__) return self.id == other.id def __repr__(self): return ''%(self.obj, self.id) def __str__(self): return ''%(self.obj,) _______________________________________________________________________________ Python 2.7.2 (default, Jul 8 2011, 23:38:53) [GCC 4.1.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> oldid=id >>> from ut.idstuff import IdHolder as id >>> from ut.idstuff import Id as idk # with caching for equal (type,value) immutables >>> oldid(2000),oldid(2000),oldid(20*100) (136189164, 136189164, 136189176) >>> id(2000),id(2000),id(20*100) # no k cacheing (, , ) >>> idk(2000),idk(2000),idk(20*100) # with k cacheing (, , ) >>> >>> oldid([]),oldid([]) # dangling pointer value returned (3083895948L, 3083895948L) >>> id([]),id([]) # pointer kept live (, ) >>> idk([]),idk([]) # pointer kept live, constant caching n/a (, ) >>> Have fun ;-) Regards Bengt Richter > > Laura = From william.leslie.ttg at gmail.com Mon Jul 11 13:30:38 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Mon, 11 Jul 2011 21:30:38 +1000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> Message-ID: On 9 July 2011 18:17, Armin Rigo wrote: > Hi, > > On Sat, Jul 9, 2011 at 5:20 AM, William ML Leslie > wrote: >> My point about small integers (...) > > I think that your point about small integers is broken (even assuming > that smallints are enabled by default, which is not the case). ?It > means that we'd get an id() function which "behaves" as long as taking > the id() of a 31-bit signed integer, and then doesn't behave as > expected neither for full 32-bit integers, nor for longs, nor for > floats. > >> ... > > I don't understand these two sentences, because they seem to say the > exact opposite of each other... For a non-smallint integer or float bound to x, "x is x" is tautological by virtue of x being represented by the same instance. There may be other objects with the same value, but I don't see why that must imply that they be the same object - why x == y must imply x is y for x and y of the same immutable type. It might make the identity dict in copy.py use slightly less memory, but it would make *much* more sense to optimise the specific use case according to the defined semantics of id(). copy.py is already effectively "broken" by the behaviour of non-cached ints on cpython; so copy.py is no excuse to break id() in pypy, which is a much more fundamental concept. >>?That does mean that id() is no longer word-sized, but it does not make it >> unbounded. > > The "unbounded" part in my e-mail was about longs. ?Obviously if you > are computing id(x) where x is in some finite set (like ints or > floats), then the results are in some finite set too. I'm not disagreeing with you there, but id has a *universally* finite range already, and imao, to change this is to make an annoying change to the language. >> Speaking of, maybe it'd be easier to attempt to get the identity dict >> into the language proper. > > We tried at some point, but python-dev refused the idea. ?Maybe the > idea has more chances for approval now that we can really show with > performance numbers that it's a deep issue, as opposed to just wave > our hands. ?Feel free to try again. ?In the meantime I've decided, at > least myself, to stick with the approach that Python is whatever is in > 2.7, and that we have to work around such issues instead of fixing > them properly in the language. Ok. It would be little help to existing code. -- William Leslie From william.leslie.ttg at gmail.com Mon Jul 11 13:36:02 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Mon, 11 Jul 2011 21:36:02 +1000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> <201107101913.p6AJDQm6027255@theraft.openend.se> Message-ID: On 11 July 2011 20:29, Bengt Richter wrote: > On 07/10/2011 09:13 PM Laura Creighton wrote: >> >> What do we want to happen when somebody -- say in a C extension -- takes >> the id of an object >> that is scheduled to be removed when the gc next runs? > > IMO taking the id should increment the object ref counter > and prevent the garbage collection, until the id value itself is garbage > collected. This significantly changes the meaning of id() in a way that will break existing code. If you want an object reference, just use one. If you want them to be persistent, build a dictionary from id to object. You can already do this yourself in pure python, and it doesn't have the side-effect of bloating id(). Otherwise, such a suggestion should go through the usual process for such a significant change to a language primitive. -- William Leslie From bokr at oz.net Mon Jul 11 15:21:21 2011 From: bokr at oz.net (Bengt Richter) Date: Mon, 11 Jul 2011 15:21:21 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> <201107101913.p6AJDQm6027255@theraft.openend.se> Message-ID: On 07/11/2011 01:36 PM William ML Leslie wrote: > On 11 July 2011 20:29, Bengt Richter wrote: >> On 07/10/2011 09:13 PM Laura Creighton wrote: >>> >>> What do we want to happen when somebody -- say in a C extension -- takes >>> the id of an object >>> that is scheduled to be removed when the gc next runs? >> >> IMO taking the id should increment the object ref counter >> and prevent the garbage collection, until the id value itself is garbage >> collected. > > This significantly changes the meaning of id() in a way that will > break existing code. > Do you have an example of existing code that depends on the integer-cast value of a dangling pointer?? Or do you mean that id's must be allowed to be compared == to integers, which my example prohibits? (I didn't define __cmp__, BTW, just lazy ;-) > If you want an object reference, just use one. If you want them to be > persistent, build a dictionary from id to object. Yes, dictionary is one way to bind an object and thus make sure its id is valid. But it would be overkill to use a dictionary to guarantee object id persistence just for the duration of an expression such as id(x.a) == id(y.a) It might be unusual, but the .a could be a property returning a dynamic value and you might be testing to see if the two return the same object, as they might if the the property get caches such values. Perhaps it is a test to verify that you have the caching version of the app. Artificial example perhaps, but current id could give false results, with id just pointing to the same dead temp space. That's an example of code that would *fail* with the current id, and be ok with id as either of Id or IdHold. As it stands, the integer returned by id persists during evaluation of an expression at least, but its validity does not necessarily last with the value even that long, as we see from the perverse (but easily explained) example: >>> id([0]) == id([1]) True >>> id([0]), id([1]) (3084230700L, 3084230700L) So at a minimum, I would think the documentation should say that an id call may return a value implicitly referencing garbage, besides hinting that there may be peculiarities about id-ing some objects. > You can already do > this yourself in pure python, and it doesn't have the side-effect of > bloating id(). My examples *are* in pure python ;-) > > Otherwise, such a suggestion should go through the usual process for > such a significant change to a language primitive. > Sure, but I only really want to understand the real (well, *intended* ;-) meaning of the id function, so I am putting forth illustrative examples to identify aspects of its current and possible behavior. Also, a new id could live alongside the old ;-) Regards, Bengt Richter > -- > William Leslie From william.leslie.ttg at gmail.com Mon Jul 11 16:21:15 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 12 Jul 2011 00:21:15 +1000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> <201107101913.p6AJDQm6027255@theraft.openend.se> Message-ID: On 11 July 2011 23:21, Bengt Richter wrote: > On 07/11/2011 01:36 PM William ML Leslie wrote: >> >> On 11 July 2011 20:29, Bengt Richter ?wrote: >>> >>> On 07/10/2011 09:13 PM Laura Creighton wrote: >>>> >>>> What do we want to happen when somebody -- say in a C extension -- takes >>>> the id of an object >>>> that is scheduled to be removed when the gc next runs? >>> >>> IMO taking the id should increment the object ref counter >>> and prevent the garbage collection, until the id value itself is garbage >>> collected. >> >> This significantly changes the meaning of id() in a way that will >> break existing code. >> > Do you have an example of existing code that depends on the integer-cast > value of a dangling pointer?? I mean that id creating a reference will break existing code. id() has always returned an integer, and the existence of some integer in some python code has never prevented some otherwise unrelated object from being collected. Existing code will not make sure that it cleans up the return value of id(), as nowhere has id() ever kept a reference to the object passed in. I know that you are suggesting that id returns something that is /not/ an integer, but that is also a language change. People have always been able to assume that they can % format ids as decimals or hexadecimals. > Or do you mean that id's must be allowed to be compared == to integers, > which my example prohibits? (I didn't define __cmp__, BTW, just lazy ;-) Good, __cmp__ has been deprecated for over 10 years now. >> If you want an object reference, just use one. ?If you want them to be >> persistent, build a dictionary from id to object. > > Yes, dictionary is one way to bind an object and thus make sure its id is > valid. > > But it would be overkill to use a dictionary to guarantee object id > persistence > just for the duration of an expression such as id(x.a) == id(y.a) But id is not about persistence. The lack of persistence is one of its key features. That said, I do think id()'s current behaviour is overkill. I just don't think we can change it in a way that will fit existing usage. And cleaning it up properly is far too much work. >> You can already do >> this yourself in pure python, and it doesn't have the side-effect of >> bloating id(). > > My examples *are* in pure python ;-) As is copy.py. We've seen several examples on this thread where you can build additional features on top of what id() gives you without changing id(). So we've no need to break id() in any of the ways that have been suggested here. >> Otherwise, such a suggestion should go through the usual process for >> such a significant change to a language primitive. >> > Sure, but I only really want to understand the real (well, *intended* ;-) > meaning of the id function, so I am putting forth illustrative examples > to identify aspects of its current and possible behavior. The notion of identity is important in any stateful language. Referential equivalence, which is a slightly more complicated (yet much better defined) idea says that x and y are equivalent when no operation can tell the difference between the two objects. 'is' is an approximation that is at least accurate for mutability of python objects. In order for x to "be" y, assignments like x.__class__ = Foo must have exactly the same effect as y.__class__ = Foo. You could presumably write a type in the implementation language that was in no way discernable from the real x, but if x is y, you *know* there is no difference. What id() does is it attempts to distil 'the thing compared' when 'is' is used. On cpython, it just returned the integer value of the pointer to the object, because on cpython that is cheap and does the job (and hey, it *is* the thing compared when you do 'is' on cpython). On pypy, things are slightly more complicated. Pypy is written in python, which has no concept of pointers. It translates to the JVM and the (safe) CLI, neither of which have a direct analogue of the pointer. And even when C or LLVM is used as the backend, objects may move around in memory. Having id() return different values after a collection cycle would be very confusing. So, pypy implements its own, quite clever mechanism for creating ids. It is described in a blog post, if you'd like to read it. The definition of id(), according to docs.python.org, is: Return the ?identity? of an object. This is an integer (or long integer) which is guaranteed to be unique and constant for this object during its lifetime. Two objects with non-overlapping lifetimes may have the same id() value. > Also, a new id could live alongside the old ;-) It's just that the problems you are attempting to fix are already solved, and they are only vaguely related to what a python programmer understands id() to mean. If, according to cpython, "1003 is not 1000 + 3", then programmers can't rely on any excellent new behaviour for id() *anyway*. OTOH, the "identity may not even be preserved for primitive types" issue is an observable difference to cpython and is fixable, even if it is a silly thing to rely on. -- William Leslie From william.leslie.ttg at gmail.com Mon Jul 11 16:23:36 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 12 Jul 2011 00:23:36 +1000 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> <201107101913.p6AJDQm6027255@theraft.openend.se> Message-ID: On 12 July 2011 00:21, William ML Leslie wrote: > Referential equivalence, which is a slightly more complicated (yet > much better defined) idea says that x and y are equivalent when no > operation can tell the difference between the two objects. Ack, sorry. I meant Referential Transparency; which is much more googleable! -- William Leslie From fijall at gmail.com Tue Jul 12 01:20:48 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 12 Jul 2011 01:20:48 +0200 Subject: [pypy-dev] Benchmarks Message-ID: Hi I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower? Current list: http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50 http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50 This is a good example why we should not work the way we work now: http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200 There was an issue, then the issue was fixed, but apparently not quite (7th of June is quite a bit slower than 25th of May) and then recently we introduced something that make it faster alltogether. Can we even fish the original issue? http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200 http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50 (is it relevant or just noise?) http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50 Cheers, fijal From anto.cuni at gmail.com Tue Jul 12 01:29:30 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 12 Jul 2011 01:29:30 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: <4E1B875A.9080702@gmail.com> On 12/07/11 01:20, Maciej Fijalkowski wrote: > Hi > > I'm a bit worried with our current benchmarks state. We have around 4 > benchmarks that had reasonable slowdowns recently and we keep putting > new features that speed up other things. How can we even say we have > actually fixed the original issue? Can we have a policy of not merging > new performance features before having a story why benchmarks got > slower? I think we really need to have support for branches on codespeed. Then, we can have a policy that a branch can be merged only if none of the benchmarks slows down. I heard that someone is working on it, but nothing concrete AFAIK, so I'm considering to do the work by myself (although I would prefer to work on something more in the core :-/) ciao, Anto From alex.gaynor at gmail.com Tue Jul 12 01:31:38 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 11 Jul 2011 16:31:38 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E1B875A.9080702@gmail.com> References: <4E1B875A.9080702@gmail.com> Message-ID: On Mon, Jul 11, 2011 at 4:29 PM, Antonio Cuni wrote: > On 12/07/11 01:20, Maciej Fijalkowski wrote: > > Hi > > > > I'm a bit worried with our current benchmarks state. We have around 4 > > benchmarks that had reasonable slowdowns recently and we keep putting > > new features that speed up other things. How can we even say we have > > actually fixed the original issue? Can we have a policy of not merging > > new performance features before having a story why benchmarks got > > slower? > > I think we really need to have support for branches on codespeed. Then, we > can > have a policy that a branch can be merged only if none of the benchmarks > slows > down. > > I heard that someone is working on it, but nothing concrete AFAIK, so I'm > considering to do the work by myself (although I would prefer to work on > something more in the core :-/) > > ciao, > Anto > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Isn't there a GSOC on that? Anyway +1 from me, if there's a regression it needs to be fixed, or reverted. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From ademan555 at gmail.com Tue Jul 12 02:51:25 2011 From: ademan555 at gmail.com (Dan Roberts) Date: Mon, 11 Jul 2011 17:51:25 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1B875A.9080702@gmail.com> Message-ID: +1 as well, but we need to make sure we can actually identify performance regressions. I don't know if currently we can draw any conclusions from significant drops in benchmark performance. +100 for benchmarking branches. I think there's a tangentially related GSoC (moving toward speed.python.org and being multi-interpreter), but I don't remember seeing a blog post announcing the accepted projects. Cheers, Dan On Jul 11, 2011 4:32 PM, "Alex Gaynor" wrote: > On Mon, Jul 11, 2011 at 4:29 PM, Antonio Cuni wrote: > >> On 12/07/11 01:20, Maciej Fijalkowski wrote: >> > Hi >> > >> > I'm a bit worried with our current benchmarks state. We have around 4 >> > benchmarks that had reasonable slowdowns recently and we keep putting >> > new features that speed up other things. How can we even say we have >> > actually fixed the original issue? Can we have a policy of not merging >> > new performance features before having a story why benchmarks got >> > slower? >> >> I think we really need to have support for branches on codespeed. Then, we >> can >> have a policy that a branch can be merged only if none of the benchmarks >> slows >> down. >> >> I heard that someone is working on it, but nothing concrete AFAIK, so I'm >> considering to do the work by myself (although I would prefer to work on >> something more in the core :-/) >> >> ciao, >> Anto >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > > Isn't there a GSOC on that? > > Anyway +1 from me, if there's a regression it needs to be fixed, or > reverted. > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Jul 12 09:01:01 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 12 Jul 2011 09:01:01 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E1B875A.9080702@gmail.com> References: <4E1B875A.9080702@gmail.com> Message-ID: On Tue, Jul 12, 2011 at 1:29 AM, Antonio Cuni wrote: > On 12/07/11 01:20, Maciej Fijalkowski wrote: >> Hi >> >> I'm a bit worried with our current benchmarks state. We have around 4 >> benchmarks that had reasonable slowdowns recently and we keep putting >> new features that speed up other things. How can we even say we have >> actually fixed the original issue? Can we have a policy of not merging >> new performance features before having a story why benchmarks got >> slower? > > I think we really need to have support for branches on codespeed. Then, we can > have a policy that a branch can be merged only if none of the benchmarks slows > down. I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well. > > I heard that someone is working on it, but nothing concrete AFAIK, so I'm > considering to do the work by myself (although I would prefer to work on > something more in the core :-/) > > ciao, > Anto > From anto.cuni at gmail.com Tue Jul 12 09:06:33 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 12 Jul 2011 09:06:33 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1B875A.9080702@gmail.com> Message-ID: <4E1BF279.6000009@gmail.com> On 12/07/11 09:01, Maciej Fijalkowski wrote: > I'll follow up on the branches, but the issue is a bit irrelevant - we > still have performance regressions by trunk checkins as well. it's not irrelevant. I won't solve the current issues, but it will help having more in the future. ciao, Anto From tobami at googlemail.com Tue Jul 12 09:16:25 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 12 Jul 2011 09:16:25 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E1BF279.6000009@gmail.com> References: <4E1B875A.9080702@gmail.com> <4E1BF279.6000009@gmail.com> Message-ID: Branches are already implemented. speed.pypy.org just needs to be upgraded to Codespeed 0.8.x and its data migrated. That has not yet been done because we wanted to move to ep.io, which has not yet happend, and we are working on speed.python.org and somehow things have stalled. Migrating current speed.pypy.org (I mean keeping it on the current server but upgrading) would not take me very long. But it is on the way out... 2011/7/12 Antonio Cuni : > On 12/07/11 09:01, Maciej Fijalkowski wrote: >> I'll follow up on the branches, but the issue is a bit irrelevant - we >> still have performance regressions by trunk checkins as well. > > it's not irrelevant. I won't solve the current issues, but it will help having > more in the future. > > ciao, > Anto > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From fijall at gmail.com Tue Jul 12 09:19:10 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 12 Jul 2011 09:19:10 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1B875A.9080702@gmail.com> <4E1BF279.6000009@gmail.com> Message-ID: On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres wrote: > Branches are already implemented. speed.pypy.org just needs to be > upgraded to Codespeed 0.8.x and its data migrated. I can help you with that, but that means having local changes from speed.pypy.org commited somewhere else. > > That has not yet been done because we wanted to move to ep.io, which > has not yet happend, and we are working on speed.python.org and > somehow things have stalled. speed.python.org is completely irrelevant here. > > Migrating current speed.pypy.org (I mean keeping it on the current > server but upgrading) would not take me very long. But it is on the > way out... We either should do the ep.io move (data has been imported up until very recently) or upgrade now. > > > > 2011/7/12 Antonio Cuni : >> On 12/07/11 09:01, Maciej Fijalkowski wrote: >>> I'll follow up on the branches, but the issue is a bit irrelevant - we >>> still have performance regressions by trunk checkins as well. >> >> it's not irrelevant. I won't solve the current issues, but it will help having >> more in the future. >> >> ciao, >> Anto >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > From anto.cuni at gmail.com Tue Jul 12 09:46:00 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 12 Jul 2011 09:46:00 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1B875A.9080702@gmail.com> <4E1BF279.6000009@gmail.com> Message-ID: <4E1BFBB8.7040808@gmail.com> On 12/07/11 09:19, Maciej Fijalkowski wrote: > On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres wrote: >> Branches are already implemented. speed.pypy.org just needs to be >> upgraded to Codespeed 0.8.x and its data migrated. oh, this is very cool, thank you :-) > I can help you with that, but that means having local changes from > speed.pypy.org commited somewhere else. what are the local changes about? > We either should do the ep.io move (data has been imported up until > very recently) or upgrade now. +1 for doing something, whatever the something is. I am willing to help, although I'm not sure what is needed. ciao, Anto From holger at merlinux.eu Tue Jul 12 09:53:48 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 12 Jul 2011 07:53:48 +0000 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1B875A.9080702@gmail.com> <4E1BF279.6000009@gmail.com> Message-ID: <20110712075348.GI12151@merlinux.eu> On Tue, Jul 12, 2011 at 09:19 +0200, Maciej Fijalkowski wrote: > On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres wrote: > > Branches are already implemented. speed.pypy.org just needs to be > > upgraded to Codespeed 0.8.x and its data migrated. > > I can help you with that, but that means having local changes from > speed.pypy.org commited somewhere else. > > > > > That has not yet been done because we wanted to move to ep.io, which > > has not yet happend, and we are working on speed.python.org and > > somehow things have stalled. > > speed.python.org is completely irrelevant here. Probably i am missing something but couldn't speed.pypy.org and speed.python.org mostly merge? (maybe except for the front page and default comparisons/settings but those could be easily hosted wherever) best, holger > > Migrating current speed.pypy.org (I mean keeping it on the current > > server but upgrading) would not take me very long. But it is on the > > way out... > > We either should do the ep.io move (data has been imported up until > very recently) or upgrade now. > > > > > > > > 2011/7/12 Antonio Cuni : > >> On 12/07/11 09:01, Maciej Fijalkowski wrote: > >>> I'll follow up on the branches, but the issue is a bit irrelevant - we > >>> still have performance regressions by trunk checkins as well. > >> > >> it's not irrelevant. I won't solve the current issues, but it will help having > >> more in the future. > >> > >> ciao, > >> Anto > >> _______________________________________________ > >> pypy-dev mailing list > >> pypy-dev at python.org > >> http://mail.python.org/mailman/listinfo/pypy-dev > >> > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From tobami at googlemail.com Tue Jul 12 10:53:52 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 12 Jul 2011 10:53:52 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <20110712075348.GI12151@merlinux.eu> References: <4E1B875A.9080702@gmail.com> <4E1BF279.6000009@gmail.com> <20110712075348.GI12151@merlinux.eu> Message-ID: They *are* being merged. The question here is to have branches *now*, as we can't know how long it will take for speed.python.org to be online. All things considered, I think the way to go would be to keep current speed.pypy.org hosting if at all possible until PyPy moves to speed.python.org. If that can be done, we can upgrade the current setup in no time. 2011/7/12 holger krekel : > On Tue, Jul 12, 2011 at 09:19 +0200, Maciej Fijalkowski wrote: >> On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres wrote: >> > Branches are already implemented. speed.pypy.org just needs to be >> > upgraded to Codespeed 0.8.x and its data migrated. >> >> I can help you with that, but that means having local changes from >> speed.pypy.org commited somewhere else. >> >> > >> > That has not yet been done because we wanted to move to ep.io, which >> > has not yet happend, and we are working on speed.python.org and >> > somehow things have stalled. >> >> speed.python.org is completely irrelevant here. > > Probably i am missing something but couldn't speed.pypy.org and > speed.python.org mostly merge? ?(maybe except for the front page and > default comparisons/settings but those could be easily hosted wherever) > > best, > holger > >> > Migrating current speed.pypy.org (I mean keeping it on the current >> > server but upgrading) would not take me very long. But it is on the >> > way out... >> >> We either should do the ep.io move (data has been imported up until >> very recently) or upgrade now. > > > >> > >> > >> > >> > 2011/7/12 Antonio Cuni : >> >> On 12/07/11 09:01, Maciej Fijalkowski wrote: >> >>> I'll follow up on the branches, but the issue is a bit irrelevant - we >> >>> still have performance regressions by trunk checkins as well. >> >> >> >> it's not irrelevant. I won't solve the current issues, but it will help having >> >> more in the future. >> >> >> >> ciao, >> >> Anto >> >> _______________________________________________ >> >> pypy-dev mailing list >> >> pypy-dev at python.org >> >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> >> > >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > From holger at merlinux.eu Tue Jul 12 13:37:39 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 12 Jul 2011 11:37:39 +0000 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1B875A.9080702@gmail.com> <4E1BF279.6000009@gmail.com> <20110712075348.GI12151@merlinux.eu> Message-ID: <20110712113737.GK12151@merlinux.eu> On Tue, Jul 12, 2011 at 10:53 +0200, Miquel Torres wrote: > They *are* being merged. The question here is to have branches *now*, > as we can't know how long it will take for speed.python.org to be > online. yes, makes sense. Better invest your time in getting branch benchmarking to work right now, it's probably the best support for pypy development ATM. > All things considered, I think the way to go would be to keep current > speed.pypy.org hosting if at all possible until PyPy moves to > speed.python.org. If that can be done, we can upgrade the current > setup in no time. This is fine by me - i am still paying for the underlying machine and so it's not fine indefinitely. OTOH people could just go to our pypy.org donation page and i could ask for some hosting costs at some point (*). best, holger (*) then again, who knows what the current global currencies will be worth in a few years or months, anyway. Better put money to good use ;) > 2011/7/12 holger krekel : > > On Tue, Jul 12, 2011 at 09:19 +0200, Maciej Fijalkowski wrote: > >> On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres wrote: > >> > Branches are already implemented. speed.pypy.org just needs to be > >> > upgraded to Codespeed 0.8.x and its data migrated. > >> > >> I can help you with that, but that means having local changes from > >> speed.pypy.org commited somewhere else. > >> > >> > > >> > That has not yet been done because we wanted to move to ep.io, which > >> > has not yet happend, and we are working on speed.python.org and > >> > somehow things have stalled. > >> > >> speed.python.org is completely irrelevant here. > > > > Probably i am missing something but couldn't speed.pypy.org and > > speed.python.org mostly merge? ?(maybe except for the front page and > > default comparisons/settings but those could be easily hosted wherever) > > > > best, > > holger > > > >> > Migrating current speed.pypy.org (I mean keeping it on the current > >> > server but upgrading) would not take me very long. But it is on the > >> > way out... > >> > >> We either should do the ep.io move (data has been imported up until > >> very recently) or upgrade now. > > > > > > > >> > > >> > > >> > > >> > 2011/7/12 Antonio Cuni : > >> >> On 12/07/11 09:01, Maciej Fijalkowski wrote: > >> >>> I'll follow up on the branches, but the issue is a bit irrelevant - we > >> >>> still have performance regressions by trunk checkins as well. > >> >> > >> >> it's not irrelevant. I won't solve the current issues, but it will help having > >> >> more in the future. > >> >> > >> >> ciao, > >> >> Anto > >> >> _______________________________________________ > >> >> pypy-dev mailing list > >> >> pypy-dev at python.org > >> >> http://mail.python.org/mailman/listinfo/pypy-dev > >> >> > >> > > >> _______________________________________________ > >> pypy-dev mailing list > >> pypy-dev at python.org > >> http://mail.python.org/mailman/listinfo/pypy-dev > >> > > > From andrew.rustytub at gmail.com Tue Jul 12 18:12:25 2011 From: andrew.rustytub at gmail.com (Andrew Evans) Date: Tue, 12 Jul 2011 09:12:25 -0700 Subject: [pypy-dev] Compiling RPython with ctypes Message-ID: Hello I am running into some snags developing an executable in pypy with translate.py and wondering what the best method for me to do this with my code would be. I am developing an Exploit Framework and one of the features I would like to have is compiling an executable. I am wondering if this is even possible with my code but it does run in the PyPy Interpreter As an example I would like to compile this code import array from ctypes import * myCode = array.array('b', "\xda\xde\xd9\x74\x24\xf4\xb8\x22\xd2\x27\x7a\x29\xc9\xb1\x4b" "\x5b\x31\x43\x1a\x83\xeb\xfc\x03\x43\x16\xe2\xd7\x3b\xbc\x7a" "\x17\xbc\x95\x4b\xd7\xd8\x92\xec\xe7\xa5\x65\x94\x08\x2d\x25" "\x69\x9d\x41\xba\xdc\x2a\xe1\xca\xf7\x25\xe2\xca\x07\xbe\xa2" "\xfe\x8a\x80\x5e\x74\xd4\x3c\xc1\x49\xb5\xb7\x91\x69\x12\x4c" "\x2c\x4e\xd1\x06\xaa\xd6\xe4\x4c\x3f\x6c\xff\x1b\x1a\x51\xfe" "\xf0\x78\xa5\x49\x8d\x4b\x4d\x48\x7f\x82\xae\x7a\xbf\x19\xfc" "\xf9\xff\x96\xfa\xc0\x30\x5b\x04\x04\x25\x90\x3d\xf6\x9d\x71" "\x37\xe7\x56\xdb\x93\xe6\x83\xba\x50\xe4\x18\xc8\x3d\xe9\x9f" "\x25\x4a\x15\x14\xb8\xa5\x9f\x6e\x9f\x29\xc1\xad\x72\x01\x53" "\xd9\x27\x5d\xac\xe6\xb1\xa5\xd2\xdc\xca\xa9\xd4\xdc\x4b\x6e" "\xd0\xdc\x4b\x71\xe0\x12\x3e\x97\xd1\x42\xd8\x57\xd6\x92\x43" "\xa9\x5c\x9c\x0d\x8e\x83\xd3\x70\xc2\x4c\x13\x73\x1b\xc4\xf6" "\x9b\x43\x29\x07\xa4\xfd\x17\x1c\xb9\xa0\x1a\x9f\x3a\xd4\xd4" "\xde\x82\xee\x16\xe0\x04\x07\xa0\x1f\xfb\x28\x26\xd1\x5f\xe6" "\x79\xbd\x0c\xf7\x2f\x39\x82\xc7\x80\xbe\xb1\xcf\xc8\xad\xc5" "\x2f\xf7\x4e\x57\xb4\x26\xf5\xdf\x51\x17\xda\x7c\xba\x39\x41" "\xf7\x9a\xb0\xfa\x92\xa8\x1a\x8f\x39\x2e\x2e\x06\xa6\x80\xf0" "\xb5\x16\x8f\x9b\x65\x78\x2e\x38\x01\xa6\x96\xe6\xe9\xc8\xb3" "\x92\xc9\x78\x53\x38\x68\xed\xcc\xcc\x05\x98\x62\x11\xb8\x06" "\xee\x38\x54\xae\x83\xce\xda\x51\x10\x40\x68\xe1\xf8\xed\xe9" "\x66\x8c\x78\x95\x58\x4e\x54\x34\xfd\xea\xaa") buffer = myCode.buffer_info()[0] my_callback = CFUNCTYPE(c_int) I would appreciate any suggestions *cheers and ty :D -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Tue Jul 12 18:30:48 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 12 Jul 2011 09:30:48 -0700 Subject: [pypy-dev] Compiling RPython with ctypes In-Reply-To: References: Message-ID: On Tue, Jul 12, 2011 at 9:12 AM, Andrew Evans wrote: > Hello I am running into some snags developing an executable in pypy with > translate.py and wondering what the best method for me to do this with my > code would be. I am developing an Exploit Framework and one of the features > I would like to have is compiling an executable. I am wondering if this is > even possible with my code but it does run in the PyPy Interpreter > > As an example I would like to compile this code > > import array > from ctypes import * > myCode = array.array('b', > "\xda\xde\xd9\x74\x24\xf4\xb8\x22\xd2\x27\x7a\x29\xc9\xb1\x4b" > > "\x5b\x31\x43\x1a\x83\xeb\xfc\x03\x43\x16\xe2\xd7\x3b\xbc\x7a" > > "\x17\xbc\x95\x4b\xd7\xd8\x92\xec\xe7\xa5\x65\x94\x08\x2d\x25" > > "\x69\x9d\x41\xba\xdc\x2a\xe1\xca\xf7\x25\xe2\xca\x07\xbe\xa2" > > "\xfe\x8a\x80\x5e\x74\xd4\x3c\xc1\x49\xb5\xb7\x91\x69\x12\x4c" > > "\x2c\x4e\xd1\x06\xaa\xd6\xe4\x4c\x3f\x6c\xff\x1b\x1a\x51\xfe" > > "\xf0\x78\xa5\x49\x8d\x4b\x4d\x48\x7f\x82\xae\x7a\xbf\x19\xfc" > > "\xf9\xff\x96\xfa\xc0\x30\x5b\x04\x04\x25\x90\x3d\xf6\x9d\x71" > > "\x37\xe7\x56\xdb\x93\xe6\x83\xba\x50\xe4\x18\xc8\x3d\xe9\x9f" > > "\x25\x4a\x15\x14\xb8\xa5\x9f\x6e\x9f\x29\xc1\xad\x72\x01\x53" > > "\xd9\x27\x5d\xac\xe6\xb1\xa5\xd2\xdc\xca\xa9\xd4\xdc\x4b\x6e" > > "\xd0\xdc\x4b\x71\xe0\x12\x3e\x97\xd1\x42\xd8\x57\xd6\x92\x43" > > "\xa9\x5c\x9c\x0d\x8e\x83\xd3\x70\xc2\x4c\x13\x73\x1b\xc4\xf6" > > "\x9b\x43\x29\x07\xa4\xfd\x17\x1c\xb9\xa0\x1a\x9f\x3a\xd4\xd4" > > "\xde\x82\xee\x16\xe0\x04\x07\xa0\x1f\xfb\x28\x26\xd1\x5f\xe6" > > "\x79\xbd\x0c\xf7\x2f\x39\x82\xc7\x80\xbe\xb1\xcf\xc8\xad\xc5" > > "\x2f\xf7\x4e\x57\xb4\x26\xf5\xdf\x51\x17\xda\x7c\xba\x39\x41" > > "\xf7\x9a\xb0\xfa\x92\xa8\x1a\x8f\x39\x2e\x2e\x06\xa6\x80\xf0" > > "\xb5\x16\x8f\x9b\x65\x78\x2e\x38\x01\xa6\x96\xe6\xe9\xc8\xb3" > > "\x92\xc9\x78\x53\x38\x68\xed\xcc\xcc\x05\x98\x62\x11\xb8\x06" > > "\xee\x38\x54\xae\x83\xce\xda\x51\x10\x40\x68\xe1\xf8\xed\xe9" > "\x66\x8c\x78\x95\x58\x4e\x54\x34\xfd\xea\xaa") > buffer = myCode.buffer_info()[0] > my_callback = CFUNCTYPE(c_int) > > I would appreciate any suggestions > > *cheers > > and ty :D > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > RPython cannot be used to translate arbitrary code, neither ctypes nor array are RPython. If you want to translate you'd need explicitly malloc the type'd memory and use rffi for external calls. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Tue Jul 12 18:34:49 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 12 Jul 2011 18:34:49 +0200 Subject: [pypy-dev] Compiling RPython with ctypes In-Reply-To: References: Message-ID: 2011/7/12 Andrew Evans > Hello I am running into some snags developing an executable in pypy with > translate.py and wondering what the best method for me to do this with my > code would be. I am developing an Exploit Framework and one of the features > I would like to have is compiling an executable. I am wondering if this is > even possible with my code but it does run in the PyPy Interpreter > ctypes is not RPython at all. But you can use the "lltype" and "rffi" modules. Your code will certainly use things like: FUNCTYPE = lltype.Ptr(lltype.FuncType([], rffi.INT)) ptr = rffi.str2charp(machineCode) # returns a "char*" pointer my_callback = rffi.cast(FUNCTYPE, ptr) my_callback() # beware! -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue Jul 12 20:01:20 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 12 Jul 2011 20:01:20 +0200 Subject: [pypy-dev] Compiling RPython with ctypes In-Reply-To: References: Message-ID: Hi Andrew, On Tue, Jul 12, 2011 at 6:34 PM, Amaury Forgeot d'Arc wrote: >> Hello I am running into some snags developing an executable in pypy with >> translate.py and wondering what the best method for me to do this with my >> code would be. In addition to the previous answer, there is also the classical "warning" answer: you are embarking on the endless (and mostly entirely pointless) fun of converting your whole program from normal Python code to RPython code. Unless that's really, *really* what you want, you should instead look at tools like py2exe, or write your custom pypy/translator/goal/app_main.py to include all your Python code pre-imported. A bient?t, Armin. From arigo at tunes.org Tue Jul 12 20:31:29 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 12 Jul 2011 20:31:29 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: Hi, On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski wrote: > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50 This was introduced by the changes we did to the GC to better support resizing big lists: 1bb155fd266f and 324a8265e420. I will look. A bient?t, Armin. From alex.gaynor at gmail.com Tue Jul 12 20:36:22 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 12 Jul 2011 11:36:22 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: I just investigated the spitfire regression, it appears to have been caused by 27df060341f0 (merg non-null-app-dict branch) Alex On Tue, Jul 12, 2011 at 11:31 AM, Armin Rigo wrote: > Hi, > > On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski > wrote: > > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50 > > This was introduced by the changes we did to the GC to better support > resizing big lists: 1bb155fd266f and 324a8265e420. I will look. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue Jul 12 20:45:44 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 12 Jul 2011 20:45:44 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: Hi Alex, On Tue, Jul 12, 2011 at 8:36 PM, Alex Gaynor wrote: > I just investigated the spitfire regression, it appears to have been caused > by 27df060341f0 (merg non-null-app-dict branch) That's not the conclusion I arrived at. The speed.pypy.org website says that revision 27df060341f0 was still fast, and I have on tannit64 a pypy-c-jit from revision 76b06820d08b which is slow. The interval contains the two changes I described in the previous e-mail, plus a few other details that are really unlikely to be the cause, but not the merge of any branch. Note that by "interval" I mean "all changes in one revision and not in the other". I'm using arigo/hack/hg/hglog here, which builds a command line like "hg log -r 'reverse(ancestors(%s) and not ancestors(%s))'". A bient?t, Armin. From alex.gaynor at gmail.com Tue Jul 12 20:50:05 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 12 Jul 2011 11:50:05 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as being slow, I narrowed it down to 45168:45205, ad there's only one reasonable commit in that range. http://paste.pocoo.org/show/437119/ are my benchmark runs (64-bit, local laptop), and the hg log which seems to correspond: http://paste.pocoo.org/show/437121/ Alex On Tue, Jul 12, 2011 at 11:45 AM, Armin Rigo wrote: > Hi Alex, > > On Tue, Jul 12, 2011 at 8:36 PM, Alex Gaynor > wrote: > > I just investigated the spitfire regression, it appears to have been > caused > > by 27df060341f0 (merg non-null-app-dict branch) > > That's not the conclusion I arrived at. The speed.pypy.org website > says that revision 27df060341f0 was still fast, and I have on tannit64 > a pypy-c-jit from revision 76b06820d08b which is slow. The interval > contains the two changes I described in the previous e-mail, plus a > few other details that are really unlikely to be the cause, but not > the merge of any branch. > > Note that by "interval" I mean "all changes in one revision and not in > the other". I'm using arigo/hack/hg/hglog here, which builds a > command line like "hg log -r 'reverse(ancestors(%s) and not > ancestors(%s))'". > > > A bient?t, > > Armin. > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue Jul 12 21:06:40 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 12 Jul 2011 21:06:40 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: Hi Alex, On Tue, Jul 12, 2011 at 8:50 PM, Alex Gaynor wrote: > speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as being > slow, I narrowed it down to 45168:45205, ad there's only one reasonable > commit in that range. The trick is that the two revisions I identify as culprit are (on tannit) revision numbers r45155 and r45156, i.e. sequentially before r45168 (which has on my repo on tannit the number r45170). We have this structure, all on the "default" branch: r45176 | r45174 / \ r45156 \ | r45170 r45155 / \ / r45154 This means that you'll miss the two revisions r45155 and r45156 if you do 45168:45205. In other words it's always subtly wrong to work with plain intervals of revision numbers in hg... The speed.pypy.org measured 45170 to be still fast, but I measured all of 45176, 45156 and 45155 to be slow. This is why I claim that the culprit is r45155. A bient?t, Armin. From alex.gaynor at gmail.com Tue Jul 12 21:10:59 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 12 Jul 2011 12:10:59 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: On Tue, Jul 12, 2011 at 12:06 PM, Armin Rigo wrote: > Hi Alex, > > On Tue, Jul 12, 2011 at 8:50 PM, Alex Gaynor > wrote: > > speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as > being > > slow, I narrowed it down to 45168:45205, ad there's only one reasonable > > commit in that range. > > The trick is that the two revisions I identify as culprit are (on > tannit) revision numbers r45155 and r45156, i.e. sequentially before > r45168 (which has on my repo on tannit the number r45170). We have > this structure, all on the "default" branch: > > r45176 > | > r45174 > / \ > r45156 \ > | r45170 > r45155 / > \ / > r45154 > > This means that you'll miss the two revisions r45155 and r45156 if you > do 45168:45205. In other words it's always subtly wrong to work with > plain intervals of revision numbers in hg... The speed.pypy.org measured > 45170 to be still fast, but I measured all of 45176, 45156 and 45155 > to be slow. This is why I claim that the culprit is r45155. > > > A bient?t, > > Armin. > Now I'm really confused, what sha does that correspond to? https://bitbucket.org/pypy/pypy/changeset/45155 seems to indicate that it is a rather boring commit on a branch? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Tue Jul 12 21:28:14 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 12 Jul 2011 12:28:14 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: Ahhhh, I found why i was getting nonsense information, `hg log -r -r` does NOT do what I wanted, you need to do `hg log -r..`! Alex On Tue, Jul 12, 2011 at 12:10 PM, Alex Gaynor wrote: > > > On Tue, Jul 12, 2011 at 12:06 PM, Armin Rigo wrote: > >> Hi Alex, >> >> On Tue, Jul 12, 2011 at 8:50 PM, Alex Gaynor >> wrote: >> > speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as >> being >> > slow, I narrowed it down to 45168:45205, ad there's only one reasonable >> > commit in that range. >> >> The trick is that the two revisions I identify as culprit are (on >> tannit) revision numbers r45155 and r45156, i.e. sequentially before >> r45168 (which has on my repo on tannit the number r45170). We have >> this structure, all on the "default" branch: >> >> r45176 >> | >> r45174 >> / \ >> r45156 \ >> | r45170 >> r45155 / >> \ / >> r45154 >> >> This means that you'll miss the two revisions r45155 and r45156 if you >> do 45168:45205. In other words it's always subtly wrong to work with >> plain intervals of revision numbers in hg... The speed.pypy.org measured >> 45170 to be still fast, but I measured all of 45176, 45156 and 45155 >> to be slow. This is why I claim that the culprit is r45155. >> >> >> A bient?t, >> >> Armin. >> > > Now I'm really confused, what sha does that correspond to? > https://bitbucket.org/pypy/pypy/changeset/45155 seems to indicate that it > is a rather boring commit on a branch? > > Alex > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Jul 12 22:04:11 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 12 Jul 2011 22:04:11 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: > > ? ? ? ?r45176 > ? ? ? ? ? | > ? ? ? ?r45174 > ? ? ? / ? ? ?\ > ?r45156 ? ? ? \ > ? ? ?| ? ? ? ?r45170 > ?r45155 ? ? ? / > ? ? ? \ ? ? ?/ > ? ? ? ?r45154 > On a completely unrelated topic, I'm always impressed how you can pull the skill to make ascii art meaningful in an email conversation :) From anto.cuni at gmail.com Wed Jul 13 00:03:31 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 13 Jul 2011 00:03:31 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: <4E1CC4B3.7000906@gmail.com> On 12/07/11 21:28, Alex Gaynor wrote: > Ahhhh, I found why i was getting nonsense information, `hg log -r > -r` does NOT do what I wanted, you need to do `hg log -r..`! uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again subtly different? From alex.gaynor at gmail.com Wed Jul 13 00:05:49 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 12 Jul 2011 15:05:49 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E1CC4B3.7000906@gmail.com> References: <4E1CC4B3.7000906@gmail.com> Message-ID: I *think* they are the same, but I really have no idea. Alex On Tue, Jul 12, 2011 at 3:03 PM, Antonio Cuni wrote: > On 12/07/11 21:28, Alex Gaynor wrote: > > Ahhhh, I found why i was getting nonsense information, `hg log -r > > -r` does NOT do what I wanted, you need to do `hg log > -r..`! > > uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again > subtly > different? > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjenvey at underboss.org Wed Jul 13 00:43:15 2011 From: pjenvey at underboss.org (Philip Jenvey) Date: Tue, 12 Jul 2011 15:43:15 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E1CC4B3.7000906@gmail.com> References: <4E1CC4B3.7000906@gmail.com> Message-ID: <7A1EAC7A-9608-4421-9DC5-CED30968947E@underboss.org> On Jul 12, 2011, at 3:03 PM, Antonio Cuni wrote: > On 12/07/11 21:28, Alex Gaynor wrote: >> Ahhhh, I found why i was getting nonsense information, `hg log -r >> -r` does NOT do what I wanted, you need to do `hg log -r..`! > > uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again subtly > different? Subtly different, hg help revset -- Philip Jenvey From alex.gaynor at gmail.com Wed Jul 13 00:47:11 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 12 Jul 2011 15:47:11 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: <7A1EAC7A-9608-4421-9DC5-CED30968947E@underboss.org> References: <4E1CC4B3.7000906@gmail.com> <7A1EAC7A-9608-4421-9DC5-CED30968947E@underboss.org> Message-ID: I don't know why I'm surprised. Alex On Tue, Jul 12, 2011 at 3:43 PM, Philip Jenvey wrote: > > On Jul 12, 2011, at 3:03 PM, Antonio Cuni wrote: > > > On 12/07/11 21:28, Alex Gaynor wrote: > >> Ahhhh, I found why i was getting nonsense information, `hg log -r > >> -r` does NOT do what I wanted, you need to do `hg log > -r..`! > > > > uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again > subtly > > different? > > Subtly different, hg help revset > > -- > Philip Jenvey > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From bokr at oz.net Wed Jul 13 11:37:01 2011 From: bokr at oz.net (Bengt Richter) Date: Wed, 13 Jul 2011 11:37:01 +0200 Subject: [pypy-dev] Object identity and dict strategies In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> <201107101913.p6AJDQm6027255@theraft.openend.se> Message-ID: On 07/11/2011 04:21 PM William ML Leslie wrote: > On 11 July 2011 23:21, Bengt Richter wrote: >> On 07/11/2011 01:36 PM William ML Leslie wrote: >>> >>> On 11 July 2011 20:29, Bengt Richter wrote: >>>> >>>> On 07/10/2011 09:13 PM Laura Creighton wrote: >>>>> >>>>> What do we want to happen when somebody -- say in a C extension -- takes >>>>> the id of an object >>>>> that is scheduled to be removed when the gc next runs? >>>> >>>> IMO taking the id should increment the object ref counter >>>> and prevent the garbage collection, until the id value itself is garbage >>>> collected. >>> >>> This significantly changes the meaning of id() in a way that will >>> break existing code. >>> >> Do you have an example of existing code that depends on the integer-cast >> value of a dangling pointer?? > > I mean that id creating a reference will break existing code. id() > has always returned an integer, and the existence of some integer in > some python code has never prevented some otherwise unrelated object > from being collected. Existing code will not make sure that it cleans > up the return value of id(), as nowhere has id() ever kept a reference > to the object passed in. Ok, d'oh ;-/ I was focused on making sure the id value "referred" to an existing live object *when returned from id* (it is of course live when passed to id while bound in id's argument -- but if that is the *only* binding, then the object is *guaranteed* to be garbage when id returns the integer, and thus that integer is IMO meaningless except as a debugging peek at implementation, and it would be an *error* for a program to depend on its value. [10:12 ~]$ python -c 'import this'|grep -A1 Errors Errors should never pass silently. Unless explicitly silenced. You are right that existing code could and some probably would break if id guarantees validity of the integer by holding the object, so I will go with the first alternative I mentioned in my reply to Armin, and focus on preventing return of the id of garbage rather than the "or else..." option which is impractical and is likely to break code, as you say. > Letting the expression result die and returning a kind of pointer > to where the result object *was* seems like a dangling pointer problem, > except I guess you can't dereference an id value (without hackery). > > Maybe id should raise an exception if the argument referenced only has > a ref count of 1 (i.e., just the reference from the argument list)? > > Or else let id be a class and return a minimal instance only binding > the passed object, and customize the compare ops to take into account > type diffs etc.? Then there would be no id values without corresponding > objects, and id values used in expressions would die a natural death, > along with their references to their objects -- whether "variables" > or expressions. > > Sorry to belabor the obvious ;-) Rather than exception, perhaps returning a None would suffice, analogous to a null pointer where no valid pointer can be returned. That should be cheap. It could also be used in answer to Laura's question, to which I only proposed the impractical id object. > > I know that you are suggesting that id returns something that is /not/ > an integer, but that is also a language change. People have always > been able to assume that they can % format ids as decimals or > hexadecimals. I thought of subclassing int, but was reaching for an id abstraction more than a practical thing, sorry. But never mind to id-as-object idea for current python ;-) > >> Or do you mean that id's must be allowed to be compared == to integers, >> which my example prohibits? (I didn't define __cmp__, BTW, just lazy ;-) > > Good, __cmp__ has been deprecated for over 10 years now. > The only sensible sort on id's I can think of off hand would be if id's carried a time stamp. >>> If you want an object reference, just use one. If you want them to be >>> persistent, build a dictionary from id to object. >> >> Yes, dictionary is one way to bind an object and thus make sure its id is >> valid. >> >> But it would be overkill to use a dictionary to guarantee object id >> persistence >> just for the duration of an expression such as id(x.a) == id(y.a) > > But id is not about persistence. The lack of persistence is one of its > key features. > > That said, I do think id()'s current behaviour is overkill. I just > don't think we can change it in a way that will fit existing usage. > And cleaning it up properly is far too much work. > How about just returning None when id sees an object which no other code will be able to see when id returns (hence making the integer the id of garbage)? > > The definition of id(), according to docs.python.org, is: > > Return the ?identity? of an object. This is an integer (or long > integer) which is guaranteed to be unique and constant for this object > during its lifetime. Two objects with non-overlapping lifetimes may > have the same id() value. > Hm, I couldn't find that, googling site:python.org Nor at site:docs.python.org. Maybe from a non-current version of docs? But never mind. >> Also, a new id could live alongside the old ;-) > > It's just that the problems you are attempting to fix are already > solved, and they are only vaguely related to what a python programmer > understands id() to mean. If, according to cpython, "1003 is not 1000 > + 3", then programmers can't rely on any excellent new behaviour for > id() *anyway*. My question to Armin was whether doing what cpython 2.7 does meant following the vagaries of possible optimizations. E.g., if space for constants were slightly modified, cpython would return False for "1003 is not 1000 +3". 1000+3 is apparently already folded to a constant 1003, but apparently local constants are currently allowed to be duplicated, as you see in in the disassembly of your example: >>> from ut.miscutil import disev >>> 1003 is not 1000 + 3 True >>> disev("1003 is not 1000 + 3") 1 0 LOAD_CONST 0 (1003) 3 LOAD_CONST 3 (1003) 6 COMPARE_OP 9 (is not) 9 RETURN_VALUE It would seem you could generate quite a few equivalent constants: >>> disev('[1000+3,1000+3,1000+3,1000+3,1000+3]') 1 0 LOAD_CONST 2 (1003) 3 LOAD_CONST 3 (1003) 6 LOAD_CONST 4 (1003) 9 LOAD_CONST 5 (1003) 12 LOAD_CONST 6 (1003) 15 BUILD_LIST 5 18 RETURN_VALUE which sooner or later someone will probably find a reason to optimize for space, and what does that mean for the *"language"* definition of id? > > OTOH, the "identity may not even be preserved for primitive types" > issue is an observable difference to cpython and is fixable, even if > it is a silly thing to rely on. > Apparently the folding of expressions yielding e.g. small integers involves generating a reference to the single instance. Hm. I downloaded pypy and it does optimize constant storage for 1003 is 1000+3 [11:03 ~]$ pypy pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:38) [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``psyco eats one brain per inch of progress'' >>>> >>>> 1003 is 1000+3 True >>>> from ut.miscutil import disev >>>> disev('1003 is 1000+3') 1 0 LOAD_CONST 0 (1003) 3 LOAD_CONST 0 (1003) 6 COMPARE_OP 8 (is) 9 RETURN_VALUE >>>> Let's see what the id values are: >>>> id(1003), id(1000+3) (-1216202084, -1216202084) >>>> disev('id(1003), id(1000+3)') 1 0 LOAD_NAME 0 (id) 3 LOAD_CONST 0 (1003) 6 CALL_FUNCTION 1 9 LOAD_NAME 0 (id) 12 LOAD_CONST 0 (1003) 15 CALL_FUNCTION 1 18 BUILD_TUPLE 2 21 RETURN_VALUE >>>> Vs cpython 2.7.2: >>> id(1003), id(1000+3) # different garbage ;-) (136814932, 136814848) >>> disev('id(1003), id(1000+3) # different garbage ;-)') 1 0 LOAD_NAME 0 (id) 3 LOAD_CONST 0 (1003) 6 CALL_FUNCTION 1 9 LOAD_NAME 0 (id) 12 LOAD_CONST 3 (1003) 15 CALL_FUNCTION 1 18 BUILD_TUPLE 2 21 RETURN_VALUE >>> Of course, the id's are all still id's of garbage locations once returned from id ;-) So how about returning None instead of id's of garbage, or raising an exception? Would that not be pythonic? Regards, Bengt Richter From arigo at tunes.org Wed Jul 13 15:42:55 2011 From: arigo at tunes.org (Armin Rigo) Date: Wed, 13 Jul 2011 15:42:55 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E1CC4B3.7000906@gmail.com> References: <4E1CC4B3.7000906@gmail.com> Message-ID: Hi, On Wed, Jul 13, 2011 at 12:03 AM, Antonio Cuni wrote: > uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again subtly > different? ...and both are different from what I find to be the *really* useful information, which is: "which checkins are contained in B but not in A"? For that you have to use -r 'ancestors(B) and not ancestors(A)', or -r 'reverse(same as before)' if you prefer the most-recent-first order. A bient?t, Armin. From sungsuha at gmail.com Wed Jul 13 16:30:21 2011 From: sungsuha at gmail.com (Seung Soo, Ha) Date: Wed, 13 Jul 2011 23:30:21 +0900 Subject: [pypy-dev] bounties for pypy In-Reply-To: References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: Has anyone considered? http://www.fossfactory.org There are a lot of projects, and the money involved makes me believe that they are worth considering. http://www.fossfactory.org/browse.php -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Wed Jul 13 16:40:13 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 13 Jul 2011 16:40:13 +0200 Subject: [pypy-dev] bounties for pypy In-Reply-To: Message from "Seung Soo, Ha" of "Wed, 13 Jul 2011 23:30:21 +0900." References: <201106281751.p5SHpixC014361@theraft.openend.se> <201106290753.p5T7rMKh002901@theraft.openend.se> <4E0EEC0C.5010809@gmail.com> Message-ID: <201107131440.p6DEeDNG012338@theraft.openend.se> In a message of Wed, 13 Jul 2011 23:30:21 +0900, "Seung Soo, Ha" writes: >--===============0151346978== >Content-Type: multipart/alternative; boundary=0016e65b5d7e9f60b104a7f4413 >2 > >--0016e65b5d7e9f60b104a7f44132 >Content-Type: text/plain; charset=UTF-8 > >Has anyone considered? > >http://www.fossfactory.org > > >There are a lot of projects, and the money involved makes me believe that >they are worth considering. >http://www.fossfactory.org/browse.php >--===============0151346978==-- They weren't one of the sites I have looked at, but I will start looking at it now. Thank you. Laura From bokr at oz.net Wed Jul 13 18:47:34 2011 From: bokr at oz.net (Bengt Richter) Date: Wed, 13 Jul 2011 18:47:34 +0200 Subject: [pypy-dev] Object identity and dict strategies a In-Reply-To: References: <20110708075857.GF20287@merlinux.eu> <4E19CE89.6070905@oz.net> <201107101913.p6AJDQm6027255@theraft.openend.se> Message-ID: On 07/13/2011 11:37 AM Bengt Richter wrote: ... > > Hm. I downloaded pypy and it does optimize constant storage for 1003 is 1000+3 > ... > > Of course, the id's are all still id's of garbage locations > once returned from id ;-) > > So how about returning None instead of id's of garbage, > or raising an exception? Would that not be pythonic? > Hm, other than practicality beating purity ;-/ Sorry to be commenting on myself, but a further thought: In a way, a constant could be considered a specially-named immutable variable, e.g., "1003" "names" 1003, so one could consider the id of an arbitrary constant (even if perhaps only "named" and its value referenced in bytecode due to constant folding from source expressions) to be the id of a live object. But I still think there will be examples of id arguments that will turn to garbage as soon as id returns -- and renders the id erroneous for any use besides a debugging peek at memory usage. I.e., there will be temp objects with no persistence beyond the scope of the argument use within id, other than while ahead in a race with garbage collection. Is there an easy way to check on whether the argument *only* has the one reference from id's arg list? When is a ref count not available? Do simple atomic constants have them? E.g., small integers vs bigger? And True, False, None? ()? and []? Regards, Bengt Richter From sdouche at gmail.com Thu Jul 14 01:21:11 2011 From: sdouche at gmail.com (Sebastien Douche) Date: Thu, 14 Jul 2011 01:21:11 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1CC4B3.7000906@gmail.com> Message-ID: On Wed, Jul 13, 2011 at 15:42, Armin Rigo wrote: > ...and both are different from what I find to be the *really* useful > information, which is: "which checkins are contained in B but not in > A"? ?For that you have to use -r 'ancestors(B) and not ancestors(A)', Not sure to understand, with Git you know easily new commits since last fetch : git log origin/master..master Same command as : git log master ^origin/master (a and not b). I think it's the same thing with hg (minus inclusion/exclusion limits). I'm wrong? -- Sebastien Douche Twitter : @sdouche From anto.cuni at gmail.com Fri Jul 15 09:17:35 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 15 Jul 2011 09:17:35 +0200 Subject: [pypy-dev] =?iso-8859-1?q?D=FCsseldorf_sprint_at_the_end_of_augus?= =?iso-8859-1?q?t=3F?= Message-ID: <4E1FE98F.7030906@gmail.com> Hi all, I know that there is a vague plan to have a sprint in D?sseldorf in the last week of august, on time with the end of Eurostars. Can we make it more concrete, decided the dates, and announce it, please? It's very likely that I'll go on vacation in the 3rd week of august, thus it would be nice to know the dates to do a proper travel plan. ciao, Anto From anto.cuni at gmail.com Fri Jul 15 10:45:04 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 15 Jul 2011 10:45:04 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: <4E1FFE10.2010807@gmail.com> On 12/07/11 01:20, Maciej Fijalkowski wrote: > Hi > > I'm a bit worried with our current benchmarks state. We have around 4 > benchmarks that had reasonable slowdowns recently [cut] Ok, let's try to make a summary of what we discovered about benchmark regressions. > > Current list: > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50 this is weird. Maybe I did something wrong (like picking the wrong revisions to test), but I cannot reproduce the slowdown. This is on my machine, 32-bit chroot: $ ./pypy-c-jit-45360-867e8ffff7a8-linux/bin/pypy ~/pypy/benchmarks/own/spectral-norm.py --take_geo_mean 0.0256132620823 $ ./pypy-c-jit-45373-582113929b62-linux/bin/pypy ~/pypy/benchmarks/own/spectral-norm.py --take_geo_mean 0.0256120378632 $ ./pypy-c-jit-45412-3476b9be3cec-linux/bin/pypy ~/pypy/benchmarks/own/spectral-norm.py --take_geo_mean 0.025725552797 on tannit, I get similar results (a bit faster than on my machine, but all the same). > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50 I think that armin investigated this, and the outcome was that it's because of the changes we did in the GC during the sprint. Armin, do you confirm? Do we have a solution? > This is a good example why we should not work the way we work now: > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200 > > There was an issue, then the issue was fixed, but apparently not quite > (7th of June is quite a bit slower than 25th of May) and then recently > we introduced something that make it faster alltogether. Can we even > fish the original issue? did anybody look at this? > http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200 I investigated this. The culprit is 5b62f71347c8, in particular the change to policy.py which enables tracing inside lltypesystem.rbuilder. With this patch, mako is fast again: http://paste.pocoo.org/show/439257/ The weird thing is that in jit-summary there is no obvious difference between the runs, apart the total time: http://paste.pocoo.org/show/439263/ Alex, do you feel like investigating more? > http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50 > (is it relevant or just noise?) > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50 I did not look into these two. Anybody? ciao, Anto From arigo at tunes.org Fri Jul 15 13:44:15 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 15 Jul 2011 13:44:15 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E1FFE10.2010807@gmail.com> References: <4E1FFE10.2010807@gmail.com> Message-ID: Hi, On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni wrote: > I think that armin investigated this, and the outcome was that it's because of > the changes we did in the GC during the sprint. Armin, do you confirm? > Do we have a solution? I confirm, and still have no solution. I tried to devise a solution (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at least one other benchmark, but this particular benchmark is still much slower than two weeks ago. A bient?t, Armin. From romain.py at gmail.com Fri Jul 15 16:58:48 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Fri, 15 Jul 2011 16:58:48 +0200 Subject: [pypy-dev] =?iso-8859-1?q?D=FCsseldorf__sprint_at_the_end_of_augu?= =?iso-8859-1?q?st=3F?= In-Reply-To: <4E1FE98F.7030906@gmail.com> References: <4E1FE98F.7030906@gmail.com> Message-ID: <20110715145848.GB7924@ubuntu> Hi Just to let you know that I will come except if the sprint occurs during the week of the 31st of August (I have an oral exam that I already postponed to attend to the Genova sprint). Cheers Romain From van.lindberg at gmail.com Fri Jul 15 19:09:30 2011 From: van.lindberg at gmail.com (VanL) Date: Fri, 15 Jul 2011 12:09:30 -0500 Subject: [pypy-dev] Sandboxing questions Message-ID: <4E20744A.8080202@gmail.com> I have a couple questions about the sandboxing feature: - Currently this is a two-process model, but early on the assertion was made that this could be done in a single process, perhaps but not necessarily separated by two OS-level threads. Is this (still?) true? What would you need to invoke to create such a pypy? - How granular can the control on imported/run functions be? Can you have a full interpreter that does everything, or an interpreter that allows socket access and that is it? Thanks, Van From fijall at gmail.com Fri Jul 15 20:50:59 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 15 Jul 2011 20:50:59 +0200 Subject: [pypy-dev] Sandboxing questions In-Reply-To: <4E20744A.8080202@gmail.com> References: <4E20744A.8080202@gmail.com> Message-ID: On Fri, Jul 15, 2011 at 7:09 PM, VanL wrote: > I have a couple questions about the sandboxing feature: > > - Currently this is a two-process model, but early on the assertion was made > that this could be done in a single process, perhaps but not necessarily > separated by two OS-level threads. Is this (still?) true? What would you > need to invoke to create such a pypy? By design, a single process thing is slightly less secure. If you say find a way to corrupt random memory, you can modify the other process, it's still only very slightly though. The sandboxing approach should work quite nicely, the hard part would be to get multiple interpreters running in a single process. It's quite a bit of work, but I would not expect it to be overly hard to do. Requires quite a bit of pypy knowledge though. > > - How granular can the control on imported/run functions be? Can you have a > full interpreter that does everything, or an interpreter that allows socket > access and that is it? It's very granular. Besides memory and CPU limits, you also control every single call that would normally be a C call, like read, write or stat, but you can implement an arbitrary custom behavior for those functions. > > Thanks, > > Van > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From van.lindberg at gmail.com Fri Jul 15 21:05:29 2011 From: van.lindberg at gmail.com (VanL) Date: Fri, 15 Jul 2011 14:05:29 -0500 Subject: [pypy-dev] Sandboxing questions In-Reply-To: References: <4E20744A.8080202@gmail.com> Message-ID: <4E208F79.1010701@gmail.com> On 7/15/2011 1:50 PM, Maciej Fijalkowski wrote: > By design, a single process thing is slightly less secure. If you say > find a way to corrupt random memory, you can modify the other process, > it's still only very slightly though. The sandboxing approach should > work quite nicely, the hard part would be to get multiple interpreters > running in a single process. It's quite a bit of work, but I would not > expect it to be overly hard to do. Requires quite a bit of pypy > knowledge though. > Could you describe a little bit more about "quite a bit of work, but... [not] overly hard to do"? What would it take, and where would someone get started? Thanks, Van From fijall at gmail.com Fri Jul 15 21:31:30 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 15 Jul 2011 21:31:30 +0200 Subject: [pypy-dev] Sandboxing questions In-Reply-To: <4E208F79.1010701@gmail.com> References: <4E20744A.8080202@gmail.com> <4E208F79.1010701@gmail.com> Message-ID: On Fri, Jul 15, 2011 at 9:05 PM, VanL wrote: > On 7/15/2011 1:50 PM, Maciej Fijalkowski wrote: >> >> By design, a single process thing is slightly less secure. If you say >> find a way to corrupt random memory, you can modify the other process, >> it's still only very slightly though. The sandboxing approach should >> work quite nicely, the hard part would be to get multiple interpreters >> running in a single process. It's quite a bit of work, but I would not >> expect it to be overly hard to do. Requires quite a bit of pypy >> knowledge though. >> > > Could you describe a little bit more about "quite a bit of work, but... > [not] overly hard to do"? What would it take, and where would someone get > started? Heh, I was kind of hoping to avoid having to answer that :-) You essentially need two things in order to achieve it: * have two interpreters in one executable (provided sandboxes don't have to be separated from each other), one constructed with sandboxing options the other without. This is something that I would describe as "run around and make it work", but probably starting with having either two copies of functions or just two copies of object spaces. * change the sandboxing transformation to call some RPython-level API instead of read/write standard output. Also provide the other end of this API. As of now the transformation walks around all graphs and changes external calls into special calls that get rendered as standard output write and standard input read. I know, this is kind of hand-waving what has to be done, I would probably start with having two interpreters in one executable, probably by having two object spaces. Cheers, fijal From fijall at gmail.com Fri Jul 15 21:31:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 15 Jul 2011 21:31:49 +0200 Subject: [pypy-dev] Sandboxing questions In-Reply-To: References: <4E20744A.8080202@gmail.com> <4E208F79.1010701@gmail.com> Message-ID: On Fri, Jul 15, 2011 at 9:31 PM, Maciej Fijalkowski wrote: > On Fri, Jul 15, 2011 at 9:05 PM, VanL wrote: >> On 7/15/2011 1:50 PM, Maciej Fijalkowski wrote: >>> >>> By design, a single process thing is slightly less secure. If you say >>> find a way to corrupt random memory, you can modify the other process, >>> it's still only very slightly though. The sandboxing approach should >>> work quite nicely, the hard part would be to get multiple interpreters >>> running in a single process. It's quite a bit of work, but I would not >>> expect it to be overly hard to do. Requires quite a bit of pypy >>> knowledge though. >>> >> >> Could you describe a little bit more about "quite a bit of work, but... >> [not] overly hard to do"? What would it take, and where would someone get >> started? > > Heh, I was kind of hoping to avoid having to answer that :-) > > You essentially need two things in order to achieve it: > > * have two interpreters in one executable (provided sandboxes don't > have to be separated from each other), one constructed with sandboxing > options the other without. This is something that I would describe as > "run around and make it work", but probably starting with having > either two copies of functions or just two copies of object spaces. > > * change the sandboxing transformation to call some RPython-level API > instead of read/write standard output. Also provide the other end of > this API. As of now the transformation walks around all graphs and > changes external calls into special calls that get rendered as > standard output write and standard input read. > > I know, this is kind of hand-waving what has to be done, I would > probably start with having two interpreters in one executable, > probably by having two object spaces. > > Cheers, > fijal > And if I may ask, what are you trying to achieve? Cheers, fijal From alex.gaynor at gmail.com Fri Jul 15 22:10:32 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 15 Jul 2011 13:10:32 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1FFE10.2010807@gmail.com> Message-ID: On Fri, Jul 15, 2011 at 4:44 AM, Armin Rigo wrote: > Hi, > > On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni > wrote: > > I think that armin investigated this, and the outcome was that it's > because of > > the changes we did in the GC during the sprint. Armin, do you confirm? > > Do we have a solution? > > I confirm, and still have no solution. I tried to devise a solution > (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at > least one other benchmark, but this particular benchmark is still much > slower than two weeks ago. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > I can own the mako one, ATM I have a bit no clue though. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From van.lindberg at gmail.com Sat Jul 16 00:32:34 2011 From: van.lindberg at gmail.com (VanL) Date: Fri, 15 Jul 2011 17:32:34 -0500 Subject: [pypy-dev] Sandboxing questions In-Reply-To: References: <4E20744A.8080202@gmail.com> <4E208F79.1010701@gmail.com> Message-ID: <4E20C002.2080601@gmail.com> On 7/15/2011 2:31 PM, Maciej Fijalkowski wrote: > > I know, this is kind of hand-waving what has to be done, I would > probably start with having two interpreters in one executable, > probably by having two object spaces. > > Cheers, > fijal > > And if I may ask, what are you trying to achieve? > Two (or more) interpreters in one executable. :) I was recently pondering the recent announcement by Armin that he thinks STM is the way to kill the GIL. I don't think the problem is the GIL; I think the problem is that we have only one. I think that a better (read: closer term, and more likely to be performant) answer is to create multiple interpreters, *each with their own GIL, each in their own thread,* and connect them via channels (essentially a pair of queues). I already knew about multiple object spaces and PyPy's sandboxing; I thought this would be the easiest way to play with that idea. Note that this is not Erlang-style processes - this is closer to appdomains (from .net), although the communications is inspired by Erlang+Go. From arigo at tunes.org Sat Jul 16 12:13:11 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 16 Jul 2011 12:13:11 +0200 Subject: [pypy-dev] Sandboxing questions In-Reply-To: <4E20C002.2080601@gmail.com> References: <4E20744A.8080202@gmail.com> <4E208F79.1010701@gmail.com> <4E20C002.2080601@gmail.com> Message-ID: Hi, On Sat, Jul 16, 2011 at 12:32 AM, VanL wrote: > I think that a better (read: closer term, and more likely to be performant) > answer is to create multiple interpreters, *each with their own GIL, each in > their own thread,* and connect them via channels (essentially a pair of > queues). That's hand-waving away the real question: what can you pass over channels? If the interpreters are supposed to be completely separated, then you can only pass strings, and the result looks exactly like separated processes. You can extend it to pass tuples and other simple data structures, but that's the same as extending the cross-process communication protocol. If on the other hand you can pass arbitrary random objects, then you have the issue that the objects are not really owned by one interpreter or the other; I don't really think it can be made to work in the current model of the object space reference. Even if we manage, we'd end up again with the issue of concurrent changes to shared objects, which is the core problem to solve in any case --- either in your approach or with STM or with fine-grained locking. A bient?t, Armin. From van.lindberg at gmail.com Sat Jul 16 22:37:46 2011 From: van.lindberg at gmail.com (VanL) Date: Sat, 16 Jul 2011 15:37:46 -0500 Subject: [pypy-dev] Sandboxing questions In-Reply-To: References: <4E20744A.8080202@gmail.com> <4E208F79.1010701@gmail.com> <4E20C002.2080601@gmail.com> Message-ID: On Jul 16, 2011 5:13 AM, "Armin Rigo" wrote: > > Hi, > > On Sat, Jul 16, 2011 at 12:32 AM, VanL wrote: > > I think that a better (read: closer term, and more likely to be performant) > > answer is to create multiple interpreters, *each with their own GIL, each in > > their own thread,* and connect them via channels (essentially a pair of > > queues). > > That's hand-waving away the real question: what can you pass over > channels? If the interpreters are supposed to be completely > separated, then you can only pass strings, and the result looks > exactly like separated processes. You can extend it to pass tuples > and other simple data structures, but that's the same as extending the > cross-process communication protocol. If on the other hand you can > pass arbitrary random objects, then you have the issue that the > objects are not really owned by one interpreter or the other; I don't > really think it can be made to work in the current model of the object > space reference. Even if we manage, we'd end up again with the issue > of concurrent changes to shared objects, which is the core problem to > solve in any case --- either in your approach or with STM or with > fine-grained locking. My intention was to proceed in four steps: First, allow the passing of any immutable type. This is about the same as multiprocessing, but you could do it without incurring the serialization/deserialization overhead. Second, allow the passing of mutable types with copy-on-write semantics. Not that this wiukd all be a sync through queues. Third, allow memory views or classes in a sending object space/thread to expose read-only access to to another objectspace /thread. The shared objects wiukd need to be explicitly declared, probably using something similar to the POSH semantics. Fourth, allow read-write access of items that were explicitly declared to be shared. One object space would be the owner of any particular object; if another object space wanted to access and modify that object, it would need to acquire the GIL for the owning object space to do so. Your STM work could eventually make acquiring the GIL for the owning object space unnecessary-but in the nearer term, I think that the semantics above would work. For example, assume objectspaces A,B, and C, each in their own thread, each with their own GIL. From the perspective of space A, B and C both look like opaque extensions. When space B wants to access something in space A, it needs to acquire GIL A. The existing GIL semantics mediate accesses to the state of space A. Part of what is interesting is that the spaces are completely independent, so you can open a socket in space A that reads and writes strings to that socket. The socket only exists in space A, so other spaces either don't see it (if it is not declared shared) or they have to acquire the GIL for space A to read or write to it. Similarly, space B can load up some modules or extensions that only exist in space B. So perhaps space A handles I/O through the socket it owns, and then sends requests/responses through the channels to spaces B...N for processing. Lets say that some of these are processor intensive; it doesn't matter. There is no shared state between the spaces/threads unless explicit synchronization is required and asked for by the programmer. You can peg one thread/space without affecting the others. Thanks, Van -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Sat Jul 16 22:44:06 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 16 Jul 2011 22:44:06 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski wrote: > Hi > > I'm a bit worried with our current benchmarks state. We have around 4 > benchmarks that had reasonable slowdowns recently and we keep putting > new features that speed up other things. How can we even say we have > actually fixed the original issue? Can we have a policy of not merging > new performance features before having a story why benchmarks got > slower? > > Current list: > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50 this fixed itself, recent runs are fast again (and anto could not reproduce at all) > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50 armin will have a look one day > > This is a good example why we should not work the way we work now: > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200 do we say "meh" and go on? > > There was an issue, then the issue was fixed, but apparently not quite > (7th of June is quite a bit slower than 25th of May) and then recently > we introduced something that make it faster alltogether. Can we even > fish the original issue? > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200 no clue so far? > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50 > (is it relevant or just noise?) just noise maybe? > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50 nobody looked > > Cheers, > fijal > From alex.gaynor at gmail.com Sat Jul 16 23:04:02 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 16 Jul 2011 14:04:02 -0700 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: On Sat, Jul 16, 2011 at 1:44 PM, Maciej Fijalkowski wrote: > On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski > wrote: > > Hi > > > > I'm a bit worried with our current benchmarks state. We have around 4 > > benchmarks that had reasonable slowdowns recently and we keep putting > > new features that speed up other things. How can we even say we have > > actually fixed the original issue? Can we have a policy of not merging > > new performance features before having a story why benchmarks got > > slower? > > > > Current list: > > > > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50 > > this fixed itself, recent runs are fast again (and anto could not > reproduce at all) > > > > > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50 > > armin will have a look one day > > > > > This is a good example why we should not work the way we work now: > > > > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200 > > do we say "meh" and go on? > > > > > There was an issue, then the issue was fixed, but apparently not quite > > (7th of June is quite a bit slower than 25th of May) and then recently > > we introduced something that make it faster alltogether. Can we even > > fish the original issue? > > > > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200 > > no clue so far? > > No clue indeed, traces don't change at all, the only change is inlining something that is never called in that code. If that causes a slowdown I claim we are broken elsewhere, and this is a symptom, not a new problem. > > > > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50 > > (is it relevant or just noise?) > > just noise maybe? > > > > > > http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50 > > nobody looked > > > > > Cheers, > > fijal > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From bokr at oz.net Sun Jul 17 11:33:18 2011 From: bokr at oz.net (Bengt Richter) Date: Sun, 17 Jul 2011 11:33:18 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: On 07/16/2011 10:44 PM Maciej Fijalkowski wrote: > On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski wrote: >> Hi >> >> I'm a bit worried with our current benchmarks state. We have around 4 >> benchmarks that had reasonable slowdowns recently and we keep putting >> new features that speed up other things. How can we even say we have >> actually fixed the original issue? Can we have a policy of not merging >> new performance features before having a story why benchmarks got >> slower? >> >> Current list: >> >> http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50 > > this fixed itself, recent runs are fast again (and anto could not > reproduce at all) > I am wondering what was sharing the tannit xeon with the the actual code being benchmarked. Are timing values stored via stdout to file(s)? How are they buffered? Could the process of allocating file storage space on a disk have reached a nonlinear overhead hump that was passed, and somehow impacted the benchmark temporarily, so it would seem to have "fixed itself?" I am thinking file system work possibly e.g. clobbering warm caches in some temporarily systematic way due to some convoying[1] between benchmark i/o and background file system stuff. Even though user time is not system time, some CPU and cache resources must be shared. Wonder what a dedicated SSD disk with large pre-allocated open files could do to normalize effects, if the disk were reformatted anew for each run? [1] http://en.wikipedia.org/wiki/Lock_convoy Regards, Bengt Richter From arigo at tunes.org Sun Jul 17 11:43:04 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 17 Jul 2011 11:43:04 +0200 Subject: [pypy-dev] Sandboxing questions In-Reply-To: References: <4E20744A.8080202@gmail.com> <4E208F79.1010701@gmail.com> <4E20C002.2080601@gmail.com> Message-ID: Hi VanL, On Sat, Jul 16, 2011 at 10:37 PM, VanL wrote: > (...) There is no shared state between the spaces/threads unless explicit > synchronization is required and asked for by the programmer. You can peg one > thread/space without affecting the others. Sure, feel free to try this out. It requires careful language-level design, like a new API with which new Python programs must be written. That's why I personally prefer my approach, because it is implementation-only, without needing any changes to existing programs. A bient?t, Armin. From fijall at gmail.com Sun Jul 17 22:15:19 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 17 Jul 2011 22:15:19 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: I think to summarize we're good now, except spitfire which is to be investigated by armin. Then new thing about go is a bit "we touched the world". Because the unoptimized traces are now shorter, less gets aborted, less gets run based on functions and it's slower. saying trace_limit=6000 makes it fast again. I guess "too bad" is the answer. From arigo at tunes.org Mon Jul 18 10:15:37 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 18 Jul 2011 10:15:37 +0200 Subject: [pypy-dev] win64 Message-ID: Hi Christian, A quick poll about the 'win64 test' branch. Is it going fine? If ll2ctypes is continuing to give you troubles, I'd suggest to move on; keep in mind that ll2ctypes is only used for testing. I would recommend that your priority should be to pass the tests in pypy/rpython/test/ and pypy/translator/c/test/, and when it is done, move on to pypy/rlib/test/ and pypy/module/*/test/. A bient?t, Armin. From tismer at stackless.com Mon Jul 18 10:30:59 2011 From: tismer at stackless.com (Christian Tismer) Date: Mon, 18 Jul 2011 10:30:59 +0200 Subject: [pypy-dev] win64 In-Reply-To: References: Message-ID: <4E23EF43.7030806@stackless.com> On 7/18/11 10:15 AM, Armin Rigo wrote: > Hi Christian, > > A quick poll about the 'win64 test' branch. Is it going fine? If > ll2ctypes is continuing to give you troubles, I'd suggest to move on; > keep in mind that ll2ctypes is only used for testing. I would > recommend that your priority should be to pass the tests in > pypy/rpython/test/ and pypy/translator/c/test/, and when it is done, > move on to pypy/rlib/test/ and pypy/module/*/test/. Hi Armin, It is going fine, ll2ctypes works since quite a while. I'm working through rpython and some things in rlib at the moment. Last 10 days I was ill with flu, that slowed me down, and I'm in big trouble with my family, divorce, finding a new apartment for me, find a way for my son to finish his school, and a lot of fight with my wife. So I can't guarantee a steady work right now, but I'm not blocked and will bring that to an end. If I get stuck, I will certainly ask you. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From anto.cuni at gmail.com Mon Jul 18 12:28:23 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 18 Jul 2011 12:28:23 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: Message-ID: <4E240AC7.9010708@gmail.com> On 17/07/11 22:15, Maciej Fijalkowski wrote: > I think to summarize we're good now, except spitfire which is to be > investigated by armin. > > Then new thing about go is a bit "we touched the world". Because the > unoptimized traces are now shorter, less gets aborted, less gets run > based on functions and it's slower. saying trace_limit=6000 makes it > fast again. I guess "too bad" is the answer. This made me wonder how effective is our "compile the loops" idea in practice, so I benchmarked translate.py first with the default settings, and then with just the function jitting: pypy ./translate.py -Ojit [Timer] Timings: [Timer] annotate --- 453.0 s [Timer] rtype_lltype --- 310.7 s [Timer] pyjitpl_lltype --- 392.6 s [Timer] backendopt_lltype --- 132.8 s [Timer] stackcheckinsertion_lltype --- 32.1 s [Timer] database_c --- 197.1 s [Timer] source_c --- 252.0 s [Timer] compile_c --- 707.6 s [Timer] =========================================== [Timer] Total: --- 2478.0 s pypy --jit threshold=-1 ./translate -Ojit [Timer] Timings: [Timer] annotate --- 486.2 s [Timer] rtype_lltype --- 297.8 s [Timer] pyjitpl_lltype --- 396.6 s [Timer] backendopt_lltype --- 128.7 s [Timer] stackcheckinsertion_lltype --- 32.6 s [Timer] database_c --- 190.0 s [Timer] source_c --- 240.0 s [Timer] compile_c --- 594.6 s [Timer] =========================================== [Timer] Total: --- 2366.5 s As you can see, if we ignore the time spent in compile_c, the total time is almost the same. What can we conclude? That "compiling the loops" is uneffective and we only care about compiling single functions? :-( ciao, Anto From arigo at tunes.org Mon Jul 18 13:58:27 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 18 Jul 2011 13:58:27 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E240AC7.9010708@gmail.com> References: <4E240AC7.9010708@gmail.com> Message-ID: Hi Anto, On Mon, Jul 18, 2011 at 12:28 PM, Antonio Cuni wrote: > What can we conclude? That "compiling the loops" is uneffective and we only > care about compiling single functions? :-( Or, conversely, that compiling single functions is ineffective and we only care about compiling the loops? No. I expect that on a large and messy program like translate.py, after a while, either approach should be fine. Still, there are cases where one or the other approach is better. If you want an obvious example where compiling loops is better, write a function that runs a loop a large number of times, but is itself called only a few times. A bient?t, Armin. From cfbolz at gmx.de Mon Jul 18 14:10:39 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 18 Jul 2011 14:10:39 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E240AC7.9010708@gmail.com> Message-ID: <4E2422BF.3090600@gmx.de> On 07/18/2011 01:58 PM, Armin Rigo wrote: > Hi Anto, > > On Mon, Jul 18, 2011 at 12:28 PM, Antonio Cuni wrote: >> What can we conclude? That "compiling the loops" is uneffective and we only >> care about compiling single functions? :-( > > Or, conversely, that compiling single functions is ineffective and we > only care about compiling the loops? No. > > I expect that on a large and messy program like translate.py, offtopic, but I still want to point out that translate is indeed terribly messy. I've been reading traces some more, and it's quite scary. E.g. we have lltype._struct.__init__ has a CALL_FUNCTION bytecode that needs more than 800 traces operations, even after optimization :-). Carl Friedrich From anto.cuni at gmail.com Mon Jul 18 14:25:15 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 18 Jul 2011 14:25:15 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E240AC7.9010708@gmail.com> Message-ID: <4E24262B.8050200@gmail.com> On 18/07/11 13:58, Armin Rigo wrote: > Or, conversely, that compiling single functions is ineffective and we > only care about compiling the loops? No. > > I expect that on a large and messy program like translate.py, after a > while, either approach should be fine. Still, there are cases where > one or the other approach is better. If you want an obvious example > where compiling loops is better, write a function that runs a loop a > large number of times, but is itself called only a few times. yes, but this was an answer to fijal's comment that "go" is now slower because the trace_limit is too high and so we compile more loops and less functions than before. I know that you can easily write examples in which one approach is much better than the other, but it's also true that usually in complex programs we are less effective than in small ones. For example, translate.py does not use much recursion, and I would expect "loop-based" compilation to be more effective in such a program. But maybe my expectation is naive, I don't know :-). ciao, Anto From anto.cuni at gmail.com Mon Jul 18 14:34:47 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 18 Jul 2011 14:34:47 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E2422BF.3090600@gmx.de> References: <4E240AC7.9010708@gmail.com> <4E2422BF.3090600@gmx.de> Message-ID: <4E242867.5080503@gmail.com> On 18/07/11 14:10, Carl Friedrich Bolz wrote: > offtopic, but I still want to point out that translate is indeed terribly > messy. I've been reading traces some more, and it's quite > scary. E.g. we have lltype._struct.__init__ has a CALL_FUNCTION > bytecode that needs more than 800 traces operations, even after > optimization :-). yes, I agree that translate.py is messy :-). Also, we have a speedup of ~2-2.5x which is more or less what you would expect by "just" removing the interpretation overhead. It probably indicates that we have laaaarge room for improvements, but I suppose that we already knew that :-) From arigo at tunes.org Mon Jul 18 15:12:47 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 18 Jul 2011 15:12:47 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E242867.5080503@gmail.com> References: <4E240AC7.9010708@gmail.com> <4E2422BF.3090600@gmx.de> <4E242867.5080503@gmail.com> Message-ID: Hi Anto, On Mon, Jul 18, 2011 at 2:34 PM, Antonio Cuni wrote: > Also, we have a speedup of ~2-2.5x which is more or less what you would expect > by "just" removing the interpretation overhead. It probably indicates that we > have laaaarge room for improvements, but I suppose that we already knew that :-) No, I would argue that it's already 2x better than "just" removing the interpretation overhead, because PyPy-no-jit is up to 2x slower than CPython to start with :-) Armin From fijall at gmail.com Mon Jul 18 21:27:40 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 18 Jul 2011 21:27:40 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E240AC7.9010708@gmail.com> Message-ID: On Mon, Jul 18, 2011 at 1:58 PM, Armin Rigo wrote: > Hi Anto, > > On Mon, Jul 18, 2011 at 12:28 PM, Antonio Cuni wrote: >> What can we conclude? That "compiling the loops" is uneffective and we only >> care about compiling single functions? :-( > > Or, conversely, that compiling single functions is ineffective and we > only care about compiling the loops? ?No. > > I expect that on a large and messy program like translate.py, after a > while, either approach should be fine. ?Still, there are cases where > one or the other approach is better. ?If you want an obvious example > where compiling loops is better, write a function that runs a loop a > large number of times, but is itself called only a few times. > > > A bient?t, > > Armin. > Or hakan's video editing code ;-) What's although worth considering is how to get stuff optimized even if we don't have loops (but I guess carl has already started) From fijall at gmail.com Mon Jul 18 21:30:01 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 18 Jul 2011 21:30:01 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: <4E242867.5080503@gmail.com> References: <4E240AC7.9010708@gmail.com> <4E2422BF.3090600@gmx.de> <4E242867.5080503@gmail.com> Message-ID: On Mon, Jul 18, 2011 at 2:34 PM, Antonio Cuni wrote: > On 18/07/11 14:10, Carl Friedrich Bolz wrote: >> offtopic, but I still want to point out that translate is indeed terribly >> messy. I've been reading traces some more, and it's quite >> scary. E.g. we have lltype._struct.__init__ has a CALL_FUNCTION >> bytecode that needs more than 800 traces operations, even after >> optimization :-). > > yes, I agree that translate.py is messy :-). > > Also, we have a speedup of ~2-2.5x which is more or less what you would expect > by "just" removing the interpretation overhead. It probably indicates that we > have laaaarge room for improvements, but I suppose that we already knew that :-) > [citation needed] I think 2x comes from 1999 or so. It must be benchmarked again to prove that. I think there are various clues that this number has changed (even maybe dropped) From tobami at googlemail.com Mon Jul 18 21:37:12 2011 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 18 Jul 2011 21:37:12 +0200 Subject: [pypy-dev] PyPy at last infinitely fast Message-ID: Hi, speed.pypy.org currently shows a very encouraging performance picture for PyPy: it is "infinite times faster than CPython". No, it is note yet April 1st. Codespeed creates the front page plots using the latest tested revision... which currently has no data for pypy-c-jit (32 bits). There is already a ticket for that issue, as well as to fix the Changes view letting you choose a revision with no data. The assumption was that a revision would be tested for all executables of a given project, which is no longer the case for PyPy. Still, even though there will be a fix, it may be a good idea to test all exes for the same revision to easy comparisons. Cheers, Miquel -------------- next part -------------- A non-text attachment was scrubbed... Name: speed_of_light.png Type: image/png Size: 51984 bytes Desc: not available URL: From anto.cuni at gmail.com Mon Jul 18 21:43:08 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 18 Jul 2011 21:43:08 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: Message-ID: <4E248CCC.6090800@gmail.com> On 18/07/11 21:37, Miquel Torres wrote: > Hi, > > speed.pypy.org currently shows a very encouraging performance picture > for PyPy: it is "infinite times faster than CPython". No, it is note > yet April 1st. hooray! We finally finished pypy :-) > Codespeed creates the front page plots using the latest tested > revision... which currently has no data for pypy-c-jit (32 bits). > There is already a ticket for that issue, as well as to fix the > Changes view letting you choose a revision with no data. The > assumption was that a revision would be tested for all executables of > a given project, which is no longer the case for PyPy. > > Still, even though there will be a fix, it may be a good idea to test > all exes for the same revision to easy comparisons. this is not completely easy, because buildbot just pulls and updates to the latest revision. If someone pushes in between the two runs, the revision is different. A quick workaround would be to force buildbot to update to a more specific revision. E.g., "the highest revision of today at 00:00", or something like this. This should ensure to have the same revision for all our benchmarks/tests. ciao, Anto From tobami at googlemail.com Mon Jul 18 22:36:04 2011 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 18 Jul 2011 22:36:04 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: <4E248CCC.6090800@gmail.com> References: <4E248CCC.6090800@gmail.com> Message-ID: Ok, it is fixed now. AND branch support is in. Results saved with branch other than "default" will be available in the comparison view. Got to talk to fijal yet to see how we should benchmark branches... Ciao! Miquel 2011/7/18 Antonio Cuni : > On 18/07/11 21:37, Miquel Torres wrote: >> Hi, >> >> speed.pypy.org currently shows a very encouraging performance picture >> for PyPy: it is "infinite times faster than CPython". No, it is note >> yet April 1st. > > hooray! We finally finished pypy :-) > >> Codespeed creates the front page plots using the latest tested >> revision... which currently has no data for pypy-c-jit (32 bits). >> There is already a ticket for that issue, as well as to fix the >> Changes view letting you choose a revision with no data. The >> assumption was that a revision would be tested for all executables of >> a given project, which is no longer the case for PyPy. >> >> Still, even though there will be a fix, it may be a good idea to test >> all exes for the same revision to easy comparisons. > > this is not completely easy, because buildbot just pulls and updates to the > latest revision. If someone pushes in between the two runs, the revision is > different. > > A quick workaround would be to force buildbot to update to a more specific > revision. E.g., "the highest revision of today at 00:00", or something like > this. This should ensure to have the same revision for all our benchmarks/tests. > > ciao, > Anto > From chef at ghum.de Mon Jul 18 23:30:26 2011 From: chef at ghum.de (Massa, Harald Armin) Date: Mon, 18 Jul 2011 23:30:26 +0200 Subject: [pypy-dev] just saw: speed.pypy.org reports 4x faster Message-ID: I recommend to wrap the code and release it with the subtitle "the 4 times faster release" Best wishes, Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Mon Jul 18 23:32:28 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 18 Jul 2011 23:32:28 +0200 Subject: [pypy-dev] just saw: speed.pypy.org reports 4x faster In-Reply-To: References: Message-ID: On Mon, Jul 18, 2011 at 11:30 PM, Massa, Harald Armin wrote: > I recommend to wrap the code and release it with the subtitle "the 4 times > faster release" I like codenames more. Like PyPy 1.6 "Kickass Panda" > Best wishes, > Harald > -- > GHUM GmbH > Harald Armin Massa > Spielberger Stra?e 49 > 70435 Stuttgart > 0173/9409607 > > Amtsgericht Stuttgart, HRB 734971 > - > persuadere. > et programmare > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > From phyo.arkarlwin at gmail.com Mon Jul 18 23:46:49 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Tue, 19 Jul 2011 04:16:49 +0630 Subject: [pypy-dev] just saw: speed.pypy.org reports 4x faster In-Reply-To: References: Message-ID: So This is really happening in 1.6 or Speed.pypy bug? On 7/19/11, Maciej Fijalkowski wrote: > On Mon, Jul 18, 2011 at 11:30 PM, Massa, Harald Armin wrote: >> I recommend to wrap the code and release it with the subtitle "the 4 times >> faster release" > > I like codenames more. Like PyPy 1.6 "Kickass Panda" > >> Best wishes, >> Harald >> -- >> GHUM GmbH >> Harald Armin Massa >> Spielberger Stra?e 49 >> 70435 Stuttgart >> 0173/9409607 >> >> Amtsgericht Stuttgart, HRB 734971 >> - >> persuadere. >> et programmare >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From tobami at googlemail.com Tue Jul 19 09:22:24 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 19 Jul 2011 09:22:24 +0200 Subject: [pypy-dev] just saw: speed.pypy.org reports 4x faster In-Reply-To: References: Message-ID: @Phyo: it's because revision 45700 shows a large improvement for telco and spitfire_stringio (40%+), which makes for an average improvement of over 6%. See detailed changes table: http://speed.pypy.org/changes/?tre=10&rev=45700%3Ac3294f3c5888&exe=1&env=1 Due to Maciej's commit (merge numpy-str-repr) or a benchmarking anomaly? Miquel 2011/7/18 Phyo Arkar : > So This is really happening in 1.6 or Speed.pypy bug? > > On 7/19/11, Maciej Fijalkowski wrote: >> On Mon, Jul 18, 2011 at 11:30 PM, Massa, Harald Armin wrote: >>> I recommend to wrap the code and release it with the subtitle "the 4 times >>> faster release" >> >> I like codenames more. Like PyPy 1.6 "Kickass Panda" >> >>> Best wishes, >>> Harald >>> -- >>> GHUM GmbH >>> Harald Armin Massa >>> Spielberger Stra?e 49 >>> 70435 Stuttgart >>> 0173/9409607 >>> >>> Amtsgericht Stuttgart, HRB 734971 >>> - >>> persuadere. >>> et programmare >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Tue Jul 19 09:32:11 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 19 Jul 2011 09:32:11 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> Message-ID: <4E2532FB.4030006@gmail.com> On 18/07/11 22:36, Miquel Torres wrote: > Ok, it is fixed now. > > AND branch support is in. Results saved with branch other than > "default" will be available in the comparison view. Got to talk to > fijal yet to see how we should benchmark branches... wow, all of this is very cool, thank you! However, there was a problem with the uploading of the results tonight :-/ http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-32/builds/791/steps/shell_7/logs/stdio Do you know what it could be? ciao, Anto From alexandervpetrov at gmail.com Wed Jul 20 08:50:20 2011 From: alexandervpetrov at gmail.com (Alexander Petrov) Date: Wed, 20 Jul 2011 09:50:20 +0300 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: <20110708003048.GA8477@ubuntu> <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> Message-ID: Thanks to all for the answers. Small remarks. I remember, I wrote such code (using itertools.repeat with slices) after spending near a day measuring performance of some syntactical constructs usually in several algorithms dealing with numbers... So I was trying to produce a sort of Python coding style for myself in that area - what constructs in what situations gave stable speed improvement and are stiil good-readable-pythonic. After that in all projects I used such things, e.g. using slice-with-repeat for assignment of constant, in prefer to for-loop or list-comprehension. Calculating primes by naive Eratosthene's sieve - is example of algorithm where a big part of calculation required to go beyong processor cache, so I suppose its performance is more characteristic of language-tools than very-low-level-compler-opmization or hardware architecture. Frankly speaking it was second piece of code that I have tested with PyPy - and suprised with such results :) Another one example I discovered where PyPy is slower is big numbers arithmetic (e.g. try to generate members of some recurrently defined sequence, such as Fibonacci, in a simple loop). =============== So at this time I didn't come to some kind of decision about PyPy. On one hand in most of the cases with straitforward code/algorithms and "common" syntax constructs there was significant speed improvement. But on the other hand, for the cases where source Python code was "optimized" or "hacked" the time of execution was sometimes better, sometimes of one order... and sometimes was a cause for this topic discussion. :) It is not bad thing generally, the bad thing that this speed degradation situations are happenned unexpectedly for me. IMO they are the most (and may be only one) interesting from the PyPy-user viewpoint. "Where and in what cases one can expect bottlenecks". Is there any documented collection of such artifacts? It can be exceptionally useful. Alex. On Sat, Jul 9, 2011 at 07:40, Alex Gaynor wrote: > I'm not too sure what could be better wrong with it, it's rather > short:?https://bitbucket.org/pypy/pypy/src/default/pypy/module/itertools/interp_itertools.py#cl-85 > Alex > > On Fri, Jul 8, 2011 at 2:18 AM, Armin Rigo wrote: >> >> Hi Alex, >> >> Before attacking the problem with the JIT, we should understand better >> why PyPy is 4-8 times slower than CPython. ?Normally you'd expect the >> factor to be at most 2. ?I suppose the answer is that our >> itertools.repeat() is bad for some reason. >> >> >> A bient?t, >> >> Armin. > > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > From davidf at sjsoft.com Wed Jul 20 09:25:20 2011 From: davidf at sjsoft.com (David Fraser) Date: Wed, 20 Jul 2011 02:25:20 -0500 (CDT) Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: Message-ID: <05cd6948-2877-4935-a6a3-c7b7ecf9689e@jackdaw.local> On Wednesday, July 20, 2011, at 8:50:20 AM, "Alexander Petrov" wrote: > [snip] > So at this time I didn't come to some kind of decision about PyPy. > > On one hand in most of the cases with straitforward code/algorithms > and "common" syntax constructs there was significant speed > improvement. > > But on the other hand, for the cases where source Python code was > "optimized" or "hacked" the time of execution was sometimes better, > sometimes of one order... and sometimes was a cause for this topic > discussion. :) It is not bad thing generally, the bad thing that this > speed degradation situations are happenned unexpectedly for me. IMO > they are the most (and may be only one) interesting from the > PyPy-user > viewpoint. "Where and in what cases one can expect bottlenecks". Is > there any documented collection of such artifacts? It can be > exceptionally useful. That's exactly what I would like. I also experimented with some simple tests and came out with PyPy being twice as slow as CPython - a wiki page which documents current areas of slowness, and potential workarounds would be fantastic - I know these things can be improved in the future, sometimes quickly, but it seems like the know-how about handling it in the mean time isn't written down anywhere... David From fijall at gmail.com Wed Jul 20 10:27:24 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Jul 2011 10:27:24 +0200 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: <05cd6948-2877-4935-a6a3-c7b7ecf9689e@jackdaw.local> References: <05cd6948-2877-4935-a6a3-c7b7ecf9689e@jackdaw.local> Message-ID: On Wed, Jul 20, 2011 at 9:25 AM, David Fraser wrote: > > > On Wednesday, July 20, 2011, at 8:50:20 AM, "Alexander Petrov" wrote: >> [snip] >> So at this time I didn't come to some kind of decision about PyPy. >> >> On one hand in most of the cases with straitforward code/algorithms >> and "common" syntax constructs there was significant speed >> improvement. >> >> But on the other hand, for the cases where source Python code was >> "optimized" or "hacked" the time of execution was sometimes better, >> sometimes of one order... and sometimes was a cause for this topic >> discussion. :) It is not bad thing generally, the bad thing that this >> speed degradation situations are happenned unexpectedly for me. IMO >> they are the most (and may be only one) interesting from the >> PyPy-user >> viewpoint. "Where and in what cases one can expect bottlenecks". Is >> there any documented collection of such artifacts? It can be >> exceptionally useful. > > That's exactly what I would like. I also experimented with some simple tests and came out with PyPy being twice as slow as CPython - a wiki page which documents current areas of slowness, and potential workarounds would be fantastic - I know these things can be improved in the future, sometimes quickly, but it seems like the know-how about handling it in the mean time isn't written down anywhere... > > David https://bitbucket.org/pypy/pypy/wiki/JitFriendliness From cfbolz at gmx.de Wed Jul 20 10:49:17 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 20 Jul 2011 10:49:17 +0200 Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: References: <20110708003048.GA8477@ubuntu> <20110708010518.3761.1343748147.divmod.xquotient.52@localhost.localdomain> Message-ID: <4E26968D.90105@gmx.de> On 07/20/2011 08:50 AM, Alexander Petrov wrote: > So at this time I didn't come to some kind of decision about PyPy. > > On one hand in most of the cases with straitforward code/algorithms > and "common" syntax constructs there was significant speed > improvement. > > But on the other hand, for the cases where source Python code was > "optimized" or "hacked" the time of execution was sometimes better, > sometimes of one order... and sometimes was a cause for this topic > discussion. :) It is not bad thing generally, the bad thing that this > speed degradation situations are happenned unexpectedly for me. IMO > they are the most (and may be only one) interesting from the PyPy-user > viewpoint. "Where and in what cases one can expect bottlenecks". Is > there any documented collection of such artifacts? It can be > exceptionally useful. Another point is that PyPy considers these programs performance bugs to be fixed. So the situation you describe will likely change in the future. Of course some perform unpredictability will always remain, that goes with a JIT somewhat. Carl Friedrich From davidf at sjsoft.com Wed Jul 20 12:36:20 2011 From: davidf at sjsoft.com (David Fraser) Date: Wed, 20 Jul 2011 05:36:20 -0500 (CDT) Subject: [pypy-dev] PyPy is much slower than CPython example / question In-Reply-To: Message-ID: On Wednesday, July 20, 2011 at 10:27:24 AM, Maciej Fijalkowski wrote: > On Wed, Jul 20, 2011 at 9:25 AM, David Fraser > wrote: > > > > > > On Wednesday, July 20, 2011, at 8:50:20 AM, "Alexander Petrov" > > wrote: > >> [snip] > >> So at this time I didn't come to some kind of decision about PyPy. > >> > >> On one hand in most of the cases with straitforward > >> code/algorithms > >> and "common" syntax constructs there was significant speed > >> improvement. > >> > >> But on the other hand, for the cases where source Python code was > >> "optimized" or "hacked" the time of execution was sometimes > >> better, > >> sometimes of one order... and sometimes was a cause for this topic > >> discussion. :) It is not bad thing generally, the bad thing that > >> this > >> speed degradation situations are happenned unexpectedly for me. > >> IMO > >> they are the most (and may be only one) interesting from the > >> PyPy-user > >> viewpoint. "Where and in what cases one can expect bottlenecks". > >> Is > >> there any documented collection of such artifacts? It can be > >> exceptionally useful. > > > > That's exactly what I would like. I also experimented with some > > simple tests and came out with PyPy being twice as slow as CPython > > - a wiki page which documents current areas of slowness, and > > potential workarounds would be fantastic - I know these things can > > be improved in the future, sometimes quickly, but it seems like > > the know-how about handling it in the mean time isn't written down > > anywhere... > > > > David > > https://bitbucket.org/pypy/pypy/wiki/JitFriendliness Fantastic, thank you... David From tobami at googlemail.com Wed Jul 20 15:56:41 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 20 Jul 2011 15:56:41 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: <4E2532FB.4030006@gmail.com> References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> Message-ID: sorry, I have currently two busy days and couldn't look yet at it. Will do as soon as I can 2011/7/19 Antonio Cuni : > On 18/07/11 22:36, Miquel Torres wrote: >> Ok, it is fixed now. >> >> AND branch support is in. Results saved with branch other than >> "default" will be available in the comparison view. Got to talk to >> fijal yet to see how we should benchmark branches... > > wow, all of this is very cool, thank you! > > However, there was a problem with the uploading of the results tonight :-/ > http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-32/builds/791/steps/shell_7/logs/stdio > > Do you know what it could be? > > ciao, > Anto > From fijall at gmail.com Wed Jul 20 16:01:42 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Jul 2011 16:01:42 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> Message-ID: On Wed, Jul 20, 2011 at 3:56 PM, Miquel Torres wrote: > sorry, I have currently two busy days and couldn't look yet at it. > Will do as soon as I can I fixed it. Having a working "changes" tab would be cool, but not immediately needed > > > 2011/7/19 Antonio Cuni : >> On 18/07/11 22:36, Miquel Torres wrote: >>> Ok, it is fixed now. >>> >>> AND branch support is in. Results saved with branch other than >>> "default" will be available in the comparison view. Got to talk to >>> fijal yet to see how we should benchmark branches... >> >> wow, all of this is very cool, thank you! >> >> However, there was a problem with the uploading of the results tonight :-/ >> http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-32/builds/791/steps/shell_7/logs/stdio >> >> Do you know what it could be? >> >> ciao, >> Anto >> > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From tobami at googlemail.com Wed Jul 20 22:01:10 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 20 Jul 2011 22:01:10 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> Message-ID: > I fixed it. Having a working "changes" tab would be cool, but not > immediately needed Thanks Maciej. It was just a case of adding the new branch parameter to the result dictionary, as you did in pypy/benchmarks/src/saveresults I wanted to save you the time to look it up and change it but well, I couldn't. Regarding the broken changes table, it is obviously a related problem, together with not making the user having to browse through empty data. All of those issues are caused by only benchmarking some of the executables with one revision, and other with different revisions. Each can be more or less solved with a couple of lines of code, but it would introduce quite a bit of overhead, so we need to consider (test/benchmark) the possible solution carefully. Could you imagine implementing Antonio's suggestion?: Citing: >A quick workaround would be to force buildbot to update to a more specific >revision. E.g., "the highest revision of today at 00:00", or something like >this. This should ensure to have the same revision for all our benchmarks/tests. Sounds good to me. Miquel 2011/7/20 Maciej Fijalkowski : > On Wed, Jul 20, 2011 at 3:56 PM, Miquel Torres wrote: >> sorry, I have currently two busy days and couldn't look yet at it. >> Will do as soon as I can > > I fixed it. Having a working "changes" tab would be cool, but not > immediately needed > >> >> >> 2011/7/19 Antonio Cuni : >>> On 18/07/11 22:36, Miquel Torres wrote: >>>> Ok, it is fixed now. >>>> >>>> AND branch support is in. Results saved with branch other than >>>> "default" will be available in the comparison view. Got to talk to >>>> fijal yet to see how we should benchmark branches... >>> >>> wow, all of this is very cool, thank you! >>> >>> However, there was a problem with the uploading of the results tonight :-/ >>> http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-32/builds/791/steps/shell_7/logs/stdio >>> >>> Do you know what it could be? >>> >>> ciao, >>> Anto >>> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > From stefan_ml at behnel.de Thu Jul 21 07:17:21 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Jul 2011 07:17:21 +0200 Subject: [pypy-dev] just saw: speed.pypy.org reports 4x faster In-Reply-To: References: Message-ID: Massa, Harald Armin, 18.07.2011 23:30: > I recommend to wrap the code and release it with the subtitle "the 4 times > faster release" Just nitpicking here, but you shouldn't forget that any given set of benchmarks can only ever be an arbitrary one. If you change the current set, you can rightfully make any claim from "PyPy is 100x faster than CPython on average" to "CPython is substantially faster than PyPy". Taking the average is a nice addition right below the graphs on speed.pypy.org, but makes no sense at all without this context. Stefan From fijall at gmail.com Thu Jul 21 09:03:16 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 21 Jul 2011 09:03:16 +0200 Subject: [pypy-dev] just saw: speed.pypy.org reports 4x faster In-Reply-To: References: Message-ID: On Thu, Jul 21, 2011 at 7:17 AM, Stefan Behnel wrote: > Massa, Harald Armin, 18.07.2011 23:30: >> >> I recommend to wrap the code and release it with the subtitle "the 4 times >> faster release" > > Just nitpicking here, but you shouldn't forget that any given set of > benchmarks can only ever be an arbitrary one. If you change the current set, > you can rightfully make any claim from "PyPy is 100x faster than CPython on > average" to "CPython is substantially faster than PyPy". Taking the average > is a nice addition right below the graphs on speed.pypy.org, but makes no > sense at all without this context. Of course :) No single number would ever describe the speed of something as complex as a python interpreter (unless the number is 42 of course). However, there were request to reduce it to that and here we have one. I don't think "4x faster now!" is a good marketing slogan even. What's significant however is that it's getting faster with each release. Cheers, fijal From anto.cuni at gmail.com Thu Jul 21 09:09:15 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 21 Jul 2011 09:09:15 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> Message-ID: <4E27D09B.8040204@gmail.com> Hi Miquel, Maciek, all, On 20/07/11 22:01, Miquel Torres wrote: > Thanks Maciej. It was just a case of adding the new branch parameter > to the result dictionary, as you did in > pypy/benchmarks/src/saveresults I think that there was another issue: currently we pass a revision number like 12345:aabbccddee, but codespeed complained that it's not a valid hg revision (it was checking that it's exactly 40 chars long). I think that fijal fixed it, not sure how. > Regarding the broken changes table, it is obviously a related problem, > together with not making the user having to browse through empty data. > All of those issues are caused by only benchmarking some of the > executables with one revision, and other with different revisions. > Each can be more or less solved with a couple of lines of code, but it > would introduce quite a bit of overhead, so we need to consider > (test/benchmark) the possible solution carefully. I think that codespeed should fix the behavior soon or later, the current one looks broken to me. However, I'm fine for having just a workaround at the moment. > Could you imagine implementing Antonio's suggestion?: > Citing: >> A quick workaround would be to force buildbot to update to a more specific >> revision. E.g., "the highest revision of today at 00:00", or something like >> this. This should ensure to have the same revision for all our benchmarks/tests. unfortunately, it's not as simple. In mercurial there is no easy way to update to a specific date/time ("hg up --date" does not consider branches, so you might end up in a different branch than default). Moreover, we want to be able to manually kick a benchmark run just for e.g. 32 bit but not on 64, so the workaround would not work in this case. I propose a new workaround: instead of having pypy-c and pypy-c-64 both in the "tannit" environment, what about having tannit-32 and tannit-64? I think this would fix the issues, at the cost of not being able to have both 32 and 64 bit plots in the same graph. What do you think? ciao, Anto From fijall at gmail.com Thu Jul 21 09:13:33 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 21 Jul 2011 09:13:33 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: <4E27D09B.8040204@gmail.com> References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> Message-ID: On Thu, Jul 21, 2011 at 9:09 AM, Antonio Cuni wrote: > Hi Miquel, Maciek, all, > > On 20/07/11 22:01, Miquel Torres wrote: > >> Thanks Maciej. It was just a case of adding the new branch parameter >> to the result dictionary, as you did in >> pypy/benchmarks/src/saveresults > > I think that there was another issue: currently we pass a revision number like > 12345:aabbccddee, but codespeed complained that it's not a valid hg revision > (it was checking that it's exactly 40 chars long). ?I think that fijal fixed > it, not sure how. by commenting out a check. Exact length is nonsense since we might pass 5 digits one day.... > >> Regarding the broken changes table, it is obviously a related problem, >> together with not making the user having to browse through empty data. >> All of those issues are caused by only benchmarking some of the >> executables with one revision, and other with different revisions. >> Each can be more or less solved with a couple of lines of code, but it >> would introduce quite a bit of overhead, so we need to consider >> (test/benchmark) the possible solution carefully. > > I think that codespeed should fix the behavior soon or later, the current one > looks broken to me. ?However, I'm fine for having just a workaround at the moment. > >> Could you imagine implementing Antonio's suggestion?: >> Citing: >>> A quick workaround would be to force buildbot to update to a more specific >>> revision. E.g., "the highest revision of today at 00:00", or something like >>> this. This should ensure to have the same revision for all our benchmarks/tests. > > unfortunately, it's not as simple. In mercurial there is no easy way to update > to a specific date/time ("hg up --date" does not consider branches, so you > might end up in a different branch than default). > > Moreover, we want to be able to manually kick a benchmark run just for e.g. 32 > bit but not on 64, so the workaround would not work in this case. > > I propose a new workaround: instead of having pypy-c and pypy-c-64 both in the > "tannit" environment, what about having tannit-32 and tannit-64? I think this > would fix the issues, at the cost of not being able to have both 32 and 64 bit > plots in the same graph. I think having them at the same graph is more important than having changes showing correct things. I might give it a go if nobody wants to > > What do you think? > > ciao, > Anto > From anto.cuni at gmail.com Thu Jul 21 09:24:55 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 21 Jul 2011 09:24:55 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> Message-ID: <4E27D447.3070807@gmail.com> On 21/07/11 09:13, Maciej Fijalkowski wrote: > I think having them at the same graph is more important than having > changes showing correct things. I might give it a go if nobody wants > to Not sure. Having them in the same graph is important only to quickly spot cases in which one backend is much slower than the other, which seems not to be the case. On the other hand, to spot regressions it's enough to have them in two separate graphs. ciao, Anto From tobami at googlemail.com Thu Jul 21 10:01:03 2011 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 21 Jul 2011 10:01:03 +0200 Subject: [pypy-dev] just saw: speed.pypy.org reports 4x faster In-Reply-To: References: Message-ID: Right. I think that the geometric average is useful mostly and primarily to gather how PyPy is improving over time, and much less to know whether it has yet reached the speed of light as compared to CPython ;-) 2011/7/21 Maciej Fijalkowski : > On Thu, Jul 21, 2011 at 7:17 AM, Stefan Behnel wrote: >> Massa, Harald Armin, 18.07.2011 23:30: >>> >>> I recommend to wrap the code and release it with the subtitle "the 4 times >>> faster release" >> >> Just nitpicking here, but you shouldn't forget that any given set of >> benchmarks can only ever be an arbitrary one. If you change the current set, >> you can rightfully make any claim from "PyPy is 100x faster than CPython on >> average" to "CPython is substantially faster than PyPy". Taking the average >> is a nice addition right below the graphs on speed.pypy.org, but makes no >> sense at all without this context. > > Of course :) > > No single number would ever describe the speed of something as complex > as a python interpreter (unless the number is 42 of course). However, > there were request to reduce it to that and here we have one. I don't > think "4x faster now!" is a good marketing slogan even. What's > significant however is that it's getting faster with each release. > > Cheers, > fijal > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From tobami at googlemail.com Thu Jul 21 10:02:57 2011 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 21 Jul 2011 10:02:57 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: <4E27D447.3070807@gmail.com> References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> Message-ID: Alternatively, the timeline view could allow to display several environments at the same time... 2011/7/21 Antonio Cuni : > On 21/07/11 09:13, Maciej Fijalkowski wrote: >> I think having them at the same graph is more important than having >> changes showing correct things. I might give it a go if nobody wants >> to > > Not sure. Having them in the same graph is important only to quickly spot > cases in which one backend is much slower than the other, which seems not to > be the case. > > On the other hand, to spot regressions it's enough to have them in two > separate graphs. > > ciao, > Anto > From anto.cuni at gmail.com Thu Jul 21 13:52:30 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 21 Jul 2011 13:52:30 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> Message-ID: <4E2812FE.8000407@gmail.com> On 21/07/11 10:02, Miquel Torres wrote: > Alternatively, the timeline view could allow to display several > environments at the same time... I think this would work. How hard is it to implement? Maciek, any opinion? ciao, Anto From fijall at gmail.com Thu Jul 21 14:11:22 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 21 Jul 2011 14:11:22 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: <4E2812FE.8000407@gmail.com> References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> Message-ID: On Thu, Jul 21, 2011 at 1:52 PM, Antonio Cuni wrote: > On 21/07/11 10:02, Miquel Torres wrote: >> Alternatively, the timeline view could allow to display several >> environments at the same time... > > I think this would work. How hard is it to implement? > Maciek, any opinion? No, not much opinion from me > > ciao, > Anto > From tobami at googlemail.com Thu Jul 21 20:45:40 2011 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 21 Jul 2011 20:45:40 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> Message-ID: Ok, not much time right now, but we will look into it. 2011/7/21 Maciej Fijalkowski : > On Thu, Jul 21, 2011 at 1:52 PM, Antonio Cuni wrote: >> On 21/07/11 10:02, Miquel Torres wrote: >>> Alternatively, the timeline view could allow to display several >>> environments at the same time... >> >> I think this would work. How hard is it to implement? >> Maciek, any opinion? > > No, not much opinion from me > >> >> ciao, >> Anto >> > From berend.deschouwer at ucs-software.co.za Fri Jul 22 12:42:31 2011 From: berend.deschouwer at ucs-software.co.za (Berend De Schouwer) Date: Fri, 22 Jul 2011 12:42:31 +0200 Subject: [pypy-dev] pypy array module memory leak? Message-ID: <4E295417.9000903@ucs-software.co.za> The following program has constant (+- 10 MB) memory usage in CPython, but it quickly leaks massive amounts of memory in pypy. It simply assigns one bit of an array to another slice. The length of the array remains constant (so there's nothing to delete from inside the program) pypy-1.5 (pypy-c-jit-43780-b590cf6de419-linux) import array import time a = array.array('i', 'a' * 100000) for i in range(1,100000): a[1:1000] = a[2001:3000] time.sleep(0.0001) # So you can go check mem usage. I apologise for the disclaimer The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes. From arigo at tunes.org Fri Jul 22 13:20:26 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 22 Jul 2011 13:20:26 +0200 Subject: [pypy-dev] pypy array module memory leak? In-Reply-To: <4E295417.9000903@ucs-software.co.za> References: <4E295417.9000903@ucs-software.co.za> Message-ID: Hi Berend, On Fri, Jul 22, 2011 at 12:42 PM, Berend De Schouwer wrote: > The following program has constant (+- 10 MB) memory usage in CPython, > but it quickly leaks massive amounts of memory in pypy. In theory it's not a leak, because the memory is eventually freed. You can see this by adding "import gc; gc.collect()" in the loop; then the memory usage remains stable. Of course in practice it's bad. The issue is that in order to trigger a collection, we count the size of the allocated objects --- but we ignore the fact that the temporary array.array created by a[2001:3000] has also an attached raw-malloced array of 1000 integers. Instead we just count the base size of the array object, which is 4 or 5 words. As a result we grossly misestimate the time at which we need to trigger the next collection. It needs to be fixed, but I'm not sure exactly how to do it generally (as opposed to just fixing array.array, which doesn't help all other similar situations). A bient?t, Armin. From Ben.Young at sungard.com Fri Jul 22 13:52:05 2011 From: Ben.Young at sungard.com (Ben.Young at sungard.com) Date: Fri, 22 Jul 2011 12:52:05 +0100 Subject: [pypy-dev] pypy array module memory leak? In-Reply-To: References: <4E295417.9000903@ucs-software.co.za> Message-ID: <01781CA2CC22B145B230504679ECF48C03A8B658@EMEA-EXCHANGE03.internal.sungard.corp> Hi Armin, .NET has an AddMemoryPressure call for just this situation. Perhaps the PyPy GCs need a similar thing? Thanks, Ben -----Original Message----- From: pypy-dev-bounces+python=theyoungfamily.co.uk at python.org [mailto:pypy-dev-bounces+python=theyoungfamily.co.uk at python.org] On Behalf Of Armin Rigo Sent: 22 July 2011 12:20 To: Berend De Schouwer Cc: pypy-dev Subject: Re: [pypy-dev] pypy array module memory leak? Hi Berend, On Fri, Jul 22, 2011 at 12:42 PM, Berend De Schouwer wrote: > The following program has constant (+- 10 MB) memory usage in CPython, > but it quickly leaks massive amounts of memory in pypy. In theory it's not a leak, because the memory is eventually freed. You can see this by adding "import gc; gc.collect()" in the loop; then the memory usage remains stable. Of course in practice it's bad. The issue is that in order to trigger a collection, we count the size of the allocated objects --- but we ignore the fact that the temporary array.array created by a[2001:3000] has also an attached raw-malloced array of 1000 integers. Instead we just count the base size of the array object, which is 4 or 5 words. As a result we grossly misestimate the time at which we need to trigger the next collection. It needs to be fixed, but I'm not sure exactly how to do it generally (as opposed to just fixing array.array, which doesn't help all other similar situations). A bient?t, Armin. _______________________________________________ pypy-dev mailing list pypy-dev at python.org http://mail.python.org/mailman/listinfo/pypy-dev From berend.deschouwer at ucs-software.co.za Fri Jul 22 14:46:02 2011 From: berend.deschouwer at ucs-software.co.za (Berend De Schouwer) Date: Fri, 22 Jul 2011 14:46:02 +0200 Subject: [pypy-dev] pypy array module memory leak? In-Reply-To: References: <4E295417.9000903@ucs-software.co.za> Message-ID: <4E29710A.6080803@ucs-software.co.za> On 22/07/2011 13:20, Armin Rigo wrote: > Hi Berend, > > On Fri, Jul 22, 2011 at 12:42 PM, Berend De Schouwer > wrote: >> The following program has constant (+- 10 MB) memory usage in CPython, >> but it quickly leaks massive amounts of memory in pypy. > In theory it's not a leak, because the memory is eventually freed. > You can see this by adding "import gc; gc.collect()" in the loop; then > the memory usage remains stable. Confirmed. It's expensive, though, so I'm running it in a separate thread. The program in question can eat a few gigs in a few seconds, so I've got to run it every second. > It needs to be fixed, but I'm not sure exactly how to do it generally > (as opposed to just fixing array.array, which doesn't help all other > similar situations). I'd appreciate pointers to fixing array.array, if possible. gc.collect() every second still eats about 500 MB too much RAM. So I'm looking at 600 vs. 100 MB RAM. At least it's running. The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes. From aidembb at yahoo.com Sat Jul 23 07:18:52 2011 From: aidembb at yahoo.com (Roger Flores) Date: Fri, 22 Jul 2011 22:18:52 -0700 (PDT) Subject: [pypy-dev] binascii.crc32() OverflowError? Message-ID: <1311398332.64833.YahooMailNeo@web45508.mail.sp1.yahoo.com> Hello all.? I'm getting an OverflowError error only when I run my program in pypy.? I've simplified it to a couple of lines: binasciiproblem.py: import binascii value = 0 new_value = 'a' value = binascii.crc32(new_value, value) & 0xffffffff value = binascii.crc32(new_value, value) & 0xffffffff Python 2.7.1: runs without problems. PyPy 1.5 and pypy-c-jit-45752-e490f90f2aa1-linux64.tar.bz2 (7/20 nightly): Traceback (most recent call last): ? File "app_main.py", line 53, in run_toplevel ? File "binasciiproblem.py", line 6, in value = binascii.crc32(new_value, value) & 0xffffffff OverflowError: long int too large to convert to int Ideas? Is this a problem in my code noticed by PyPy or a PyPy library API issue?? I've been toying around with a PPM compressor and could sure use PyPy's speed boosts for the longer runs! Thanks, -Roger From arigo at tunes.org Sat Jul 23 10:58:53 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 23 Jul 2011 10:58:53 +0200 Subject: [pypy-dev] binascii.crc32() OverflowError? In-Reply-To: <1311398332.64833.YahooMailNeo@web45508.mail.sp1.yahoo.com> References: <1311398332.64833.YahooMailNeo@web45508.mail.sp1.yahoo.com> Message-ID: Hi, On Sat, Jul 23, 2011 at 7:18 AM, Roger Flores wrote: > value = binascii.crc32(new_value, value) & 0xffffffff > value = binascii.crc32(new_value, value) & 0xffffffff Thanks for the report. This code doesn't run on top of CPython 2.5, too, but works indeed on top of CPython 2.7. We already had troubles deciding the type of the return value of crc32(), as it's not consistent on CPython. Now I see that we have troubles with the argument too :-) It seems that crc32() and a few similar functions actually accept integers of *any* size as argument, and just truncate it. I'll just fix it in PyPy to also accept integers of any size. A bient?t, Armin. From arigo at tunes.org Sat Jul 23 11:05:12 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 23 Jul 2011 11:05:12 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E240AC7.9010708@gmail.com> Message-ID: Hi Maciek, On Mon, Jul 18, 2011 at 9:27 PM, Maciej Fijalkowski wrote: > What's although worth considering is how to get stuff optimized even > if we don't have loops (but I guess carl has already started) I'm unsure what you mean here. The function_threshold stuff you did is exactly that, no? A bient?t, Armin. From fijall at gmail.com Sat Jul 23 11:10:31 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 23 Jul 2011 11:10:31 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E240AC7.9010708@gmail.com> Message-ID: On Sat, Jul 23, 2011 at 11:05 AM, Armin Rigo wrote: > Hi Maciek, > > On Mon, Jul 18, 2011 at 9:27 PM, Maciej Fijalkowski wrote: >> What's although worth considering is how to get stuff optimized even >> if we don't have loops (but I guess carl has already started) > > I'm unsure what you mean here. ?The function_threshold stuff you did > is exactly that, no? > Hey. No, I meant something else. Can you show up on IRC to discuss? :) > > A bient?t, > > Armin. > From fijall at gmail.com Sat Jul 23 12:39:15 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 23 Jul 2011 12:39:15 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E240AC7.9010708@gmail.com> Message-ID: On Sat, Jul 23, 2011 at 11:05 AM, Armin Rigo wrote: > Hi Maciek, > > On Mon, Jul 18, 2011 at 9:27 PM, Maciej Fijalkowski wrote: >> What's although worth considering is how to get stuff optimized even >> if we don't have loops (but I guess carl has already started) > > I'm unsure what you mean here. ?The function_threshold stuff you did > is exactly that, no? > A verbose answer - the function_threshold is about it, but also the optimization level is much lower if we can't do loop invariant code motion. One example is global lookups, where carl (or his student) is working on eliminating guards even if we don't do LICM. This is what I meant by "optimizing" a no-loop scenario. > > A bient?t, > > Armin. > From arigo at tunes.org Sat Jul 23 22:41:03 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 23 Jul 2011 22:41:03 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E240AC7.9010708@gmail.com> Message-ID: Hi, On Sat, Jul 23, 2011 at 12:39 PM, Maciej Fijalkowski wrote: > A verbose answer - the function_threshold is about it, but also the > optimization level is much lower if we can't do loop invariant code > motion. One example is global lookups, where carl (or his student) is > working on eliminating guards even if we don't do LICM. This is what I > meant by "optimizing" a no-loop scenario. Advanced. :-) Armin From fijall at gmail.com Sun Jul 24 10:20:01 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 24 Jul 2011 10:20:01 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1FFE10.2010807@gmail.com> Message-ID: On Sun, Jul 24, 2011 at 10:12 AM, Armin Rigo wrote: > Hi all, > > On Fri, Jul 15, 2011 at 1:44 PM, Armin Rigo wrote: >> On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni wrote: >>> I think that armin investigated this, and the outcome was that it's because of >>> the changes we did in the GC during the sprint. Armin, do you confirm? >>> Do we have a solution? >> >> I confirm, and still have no solution. ?I tried to devise a solution >> (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at >> least one other benchmark, but this particular benchmark is still much >> slower than two weeks ago. > > I found and fixed the bug in 82e5051d55c3 (and, to a less major > extend, in 6b4eb34c6091). ?I also did 4c0d2555caa8: generate from the > jit backend assembler code to set the GC card bit inline, using just > 4-5 instructions, instead of in the write barrier call. ?Finally I > reverted PYPY_GC_LOSTCARD because it sounds like a hack and isn't > really compatible with 4c0d2555caa8. ?All in all it gives good > results. > > I can also confirm that both chaos in 3f7e and go in e911 are randomly > slowed down by jit tracer improvements, and would become fast again > with a smaller --jit trace_limit... > That probably should be addressed (maybe?) so can we decide we're done with regressions? Cheers, fijal From arigo at tunes.org Sun Jul 24 10:12:47 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 24 Jul 2011 10:12:47 +0200 Subject: [pypy-dev] Benchmarks In-Reply-To: References: <4E1FFE10.2010807@gmail.com> Message-ID: Hi all, On Fri, Jul 15, 2011 at 1:44 PM, Armin Rigo wrote: > On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni wrote: >> I think that armin investigated this, and the outcome was that it's because of >> the changes we did in the GC during the sprint. Armin, do you confirm? >> Do we have a solution? > > I confirm, and still have no solution. ?I tried to devise a solution > (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at > least one other benchmark, but this particular benchmark is still much > slower than two weeks ago. I found and fixed the bug in 82e5051d55c3 (and, to a less major extend, in 6b4eb34c6091). I also did 4c0d2555caa8: generate from the jit backend assembler code to set the GC card bit inline, using just 4-5 instructions, instead of in the write barrier call. Finally I reverted PYPY_GC_LOSTCARD because it sounds like a hack and isn't really compatible with 4c0d2555caa8. All in all it gives good results. I can also confirm that both chaos in 3f7e and go in e911 are randomly slowed down by jit tracer improvements, and would become fast again with a smaller --jit trace_limit... A bient?t, Armin. From anto.cuni at gmail.com Sun Jul 24 19:06:10 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 24 Jul 2011 19:06:10 +0200 Subject: [pypy-dev] [pypy-commit] pypy default: signal.signal() crashes with ValueError when called from a In-Reply-To: <20110724164707.DB19A8208C@wyvern.cs.uni-duesseldorf.de> References: <20110724164707.DB19A8208C@wyvern.cs.uni-duesseldorf.de> Message-ID: <4E2C5102.6090207@gmail.com> Hi Armin, On 24/07/11 18:47, arigo wrote: > Author: Armin Rigo > Branch: > Changeset: r45943:824b72bb6b45 > Date: 2011-07-24 17:25 +0200 > http://bitbucket.org/pypy/pypy/changeset/824b72bb6b45/ > > Log: signal.signal() crashes with ValueError when called from a non-main > thread...? > > diff --git a/lib_pypy/pyrepl/unix_console.py b/lib_pypy/pyrepl/unix_console.py [cut] Note that pyrepl exists as an independent repo and that the one in PyPy is supposed to be a copy of it (similar to what we do in py.test): https://bitbucket.org/pypy/pyrepl For example, it is automatically installed by pdb++ on top of cpython to have the colored tab completion. ciao, Anto From berend.deschouwer at ucs-software.co.za Mon Jul 25 08:59:18 2011 From: berend.deschouwer at ucs-software.co.za (Berend De Schouwer) Date: Mon, 25 Jul 2011 08:59:18 +0200 Subject: [pypy-dev] pypy array module memory leak? In-Reply-To: <4E295417.9000903@ucs-software.co.za> References: <4E295417.9000903@ucs-software.co.za> Message-ID: <4E2D1446.50606@ucs-software.co.za> I've replaced a = array.array() with a = [0] * 10000 and it's faster, and doesn't eat all the RAM. For those who care :) On 22/07/2011 12:42, Berend De Schouwer wrote: > The following program has constant (+- 10 MB) memory usage in CPython, > but it quickly leaks massive amounts of memory in pypy. > > It simply assigns one bit of an array to another slice. The length of > the array remains constant (so there's nothing to delete from inside the > program) > > pypy-1.5 (pypy-c-jit-43780-b590cf6de419-linux) > > > import array > import time > > a = array.array('i', 'a' * 100000) > for i in range(1,100000): > a[1:1000] = a[2001:3000] > time.sleep(0.0001) # So you can go check mem usage. > > > I apologise for the disclaimer > > > > The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes. From lac at openend.se Mon Jul 25 09:38:44 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 25 Jul 2011 09:38:44 +0200 Subject: [pypy-dev] Bitbucket offering to donate servers, hosting ... Message-ID: <201107250738.p6P7ci4X032329@theraft.openend.se> This showed up on the infrastructure mailing list. I'd like a really nice windows machine always available for benchmarking and another one to test on. What would other people like? Make a list and we can ask Charles McLaughlin if the offer is still open. As far as I know the OSU/OSL hosting is linux only. Laura (still reading 10 days of email) ------- Forwarded Message Return-Path: infrastructure-bounces+lac=openend.se at python.org Delivery-Date: Fri Jul 22 20:05:08 2011 Date: Tue, 19 Jul 2011 07:08:31 -0700 From: Aahz To: infrastructure at python.org, psf at python.org Also cc'ing the board due to the recent discussion about OSU/OSL. I figured it wasn't my place to tell Charles about that. In case it's not clear from the e-mail address, it appears that Bitbucket was acquired by Atlassian. One idea: maybe Atlassian/Bitbucket would be willing to donate hardware to the OSU/OSL hosting, as long as we give them a nice public thanks. - ----- Forwarded message from Aahz ----- > Date: Tue, 19 Jul 2011 07:03:49 -0700 > From: Aahz > To: Charles McLaughlin > Cc: pydotorg-www at python.org > Subject: Re: [pydotorg-www] Volunteer > Organization: The Cat & Dragon > > On Mon, Jul 18, 2011, Charles McLaughlin wrote: >> >> I noticed python.org was down a couple times recently and I wondered if you >> guys need any more volunteers to help with infrastructure. I'm the sysadmin >> at bitbucket.org. No guarantees without knowing what I'm getting myself >> into, but in addition to some of my time we might also be able to donate >> servers, hosting, bandwidth, etc if that would help. > > Thanks for the offer, we already have good hosting donated by XS4ALL. > > Volunteer time is of course more precious and we never have enough. I > think probably the best place to discuss your offer would be the > Infrastructure Committee, I'm forwarding it to them. You can also reach > them directly at infrastructure at python.org > -- > Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ > > "If you don't know what your program is supposed to do, you'd better not > start writing it." --Dijkstra > _______________________________________________ > pydotorg-www mailing list > pydotorg-www at python.org > http://mail.python.org/mailman/listinfo/pydotorg-www - ----- End forwarded message ----- - -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "If you don't know what your program is supposed to do, you'd better not start writing it." --Dijkstra ________________________________________________ Infrastructure mailing list Infrastructure at python.org http://mail.python.org/mailman/listinfo/infrastructure Unsubscribe: http://mail.python.org/mailman/options/infrastructure/lac%40openend.se ------- End of Forwarded Message From ian at ianozsvald.com Mon Jul 25 11:00:28 2011 From: ian at ianozsvald.com (Ian Ozsvald) Date: Mon, 25 Jul 2011 10:00:28 +0100 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) Message-ID: Dear all, I've published v0.2 of my High Performance Python tutorial write-up from the session I ran at EuroPython: http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from-europython-2011/ Antonio - you asked earlier if the 'expanded math' version of the Mandelbrot solver (using doubles rather than complex numbers) would be faster - I've timed it and it is a bit faster with a nightly build of PyPy, but nowhere near as fast at ShedSkin's generated C output (details below). Maciej - thanks for pointing me at the numpy module. I've added a tiny section showing numpy in PyPy but I haven't converted the Mandelbrot solver to use it (even finishing v0.2 took longer than I'd thought). I'm hoping that some more exposure in the report might bring in more volunteers from outside. Here's a clip from the report in the PyPy section: "By running pypy pure_python.py 1000 1000 on my MacBook it takes 5.9 seconds, running pypy pure_python_2.py 1000 1000 takes 4.9 seconds. (Ian - the only difference with pure_python_2.py is that local dereferences in the tight loop are moved outside the loop, causing fewer dereference operations) As an additional test (not shown in the graphs) I ran pypy shedskin2.py 1000 1000 which runs the expanded math version of the shedskin variant below (this replaces complex numbers with floats and expands abs to avoid the square root). The shedskin2.py result takes 3.2 seconds (which is still much slower than the 0.4s version compiled using shedskin)." The pure_python src is here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/master/mandelbrot/python shedskin2.py is available here: https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/master/mandelbrot/shedskin I haven't tested whether the warm-up periods for PyPy are significant, possibly they account for much of the difference between ShedSkin and PyPy? I want to revisit this but for the next few weeks I have to go back to other projects. I hope the report brings in some new folk for PyPy, Ian. -- Ian Ozsvald (A.I. researcher, screencaster) ian at IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald From fijall at gmail.com Mon Jul 25 11:08:12 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Jul 2011 11:08:12 +0200 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: References: Message-ID: On Mon, Jul 25, 2011 at 11:00 AM, Ian Ozsvald wrote: > Dear all, I've published v0.2 of my High Performance Python tutorial > write-up from the session I ran at EuroPython: > http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from-europython-2011/ > > Antonio - you asked earlier if the 'expanded math' version of the > Mandelbrot solver (using doubles rather than complex numbers) would be > faster - I've timed it and it is a bit faster with a nightly build of > PyPy, but nowhere near as fast at ShedSkin's generated C output > (details below). > > Maciej - thanks for pointing me at the numpy module. I've added a tiny > section showing numpy in PyPy but I haven't converted the Mandelbrot > solver to use it (even finishing v0.2 took longer than I'd thought). > I'm hoping that some more exposure in the report might bring in more > volunteers from outside. > > Here's a clip from the report in the PyPy section: > "By running pypy pure_python.py 1000 1000 on my MacBook it takes 5.9 > seconds, running pypy pure_python_2.py 1000 1000 takes 4.9 seconds. > (Ian - the only difference with pure_python_2.py is that local > dereferences in the tight loop are moved outside the loop, causing > fewer dereference operations) > > As an additional test (not shown in the graphs) I ran pypy > shedskin2.py 1000 1000 which runs the expanded math version of the > shedskin variant below (this replaces complex numbers with floats and > expands abs to avoid the square root). The shedskin2.py result takes > 3.2 seconds (which is still much slower than the 0.4s version compiled > using shedskin)." > > The pure_python src is here: > https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/master/mandelbrot/python > > shedskin2.py is available here: > https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/master/mandelbrot/shedskin > > I haven't tested whether the warm-up periods for PyPy are significant, > possibly they account for much of the difference between ShedSkin and > PyPy? I want to revisit this but for the next few weeks I have to go > back to other projects. Most come from the fact that you're using lists and not say array.array (or numpy array), so the storage is not optimized. ShedSkin doesn't allow you to store different types in a list. We'll make it fast one day even if you use list, but indeed, using array.array would make it much faster. Cheers, fijal From ian at ianozsvald.com Mon Jul 25 12:42:13 2011 From: ian at ianozsvald.com (Ian Ozsvald) Date: Mon, 25 Jul 2011 11:42:13 +0100 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: References: Message-ID: Ah! Ok, I've noted array.array (along with running bigger tests to check for JIT warm-up overheads). Hopefully I'll get some more time in a few weeks to play with variants. Cheers! i. On 25 July 2011 10:08, Maciej Fijalkowski wrote: > On Mon, Jul 25, 2011 at 11:00 AM, Ian Ozsvald wrote: >> Dear all, I've published v0.2 of my High Performance Python tutorial >> write-up from the session I ran at EuroPython: >> http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from-europython-2011/ >> >> Antonio - you asked earlier if the 'expanded math' version of the >> Mandelbrot solver (using doubles rather than complex numbers) would be >> faster - I've timed it and it is a bit faster with a nightly build of >> PyPy, but nowhere near as fast at ShedSkin's generated C output >> (details below). >> >> Maciej - thanks for pointing me at the numpy module. I've added a tiny >> section showing numpy in PyPy but I haven't converted the Mandelbrot >> solver to use it (even finishing v0.2 took longer than I'd thought). >> I'm hoping that some more exposure in the report might bring in more >> volunteers from outside. >> >> Here's a clip from the report in the PyPy section: >> "By running pypy pure_python.py 1000 1000 on my MacBook it takes 5.9 >> seconds, running pypy pure_python_2.py 1000 1000 takes 4.9 seconds. >> (Ian - the only difference with pure_python_2.py is that local >> dereferences in the tight loop are moved outside the loop, causing >> fewer dereference operations) >> >> As an additional test (not shown in the graphs) I ran pypy >> shedskin2.py 1000 1000 which runs the expanded math version of the >> shedskin variant below (this replaces complex numbers with floats and >> expands abs to avoid the square root). The shedskin2.py result takes >> 3.2 seconds (which is still much slower than the 0.4s version compiled >> using shedskin)." >> >> The pure_python src is here: >> https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/master/mandelbrot/python >> >> shedskin2.py is available here: >> https://github.com/ianozsvald/EuroPython2011_HighPerformanceComputing/tree/master/mandelbrot/shedskin >> >> I haven't tested whether the warm-up periods for PyPy are significant, >> possibly they account for much of the difference between ShedSkin and >> PyPy? I want to revisit this but for the next few weeks I have to go >> back to other projects. > > Most come from the fact that you're using lists and not say > array.array (or numpy array), so the storage is not optimized. > ShedSkin doesn't allow you to store different types in a list. We'll > make it fast one day even if you use list, but indeed, using > array.array would make it much faster. > > Cheers, > fijal > -- Ian Ozsvald (A.I. researcher, screencaster) ian at IanOzsvald.com http://IanOzsvald.com http://SocialTiesApp.com/ http://MorConsulting.com/ http://blog.AICookbook.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald From arigo at tunes.org Mon Jul 25 13:38:51 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Jul 2011 13:38:51 +0200 Subject: [pypy-dev] [pypy-commit] pypy default: signal.signal() crashes with ValueError when called from a In-Reply-To: <4E2C5102.6090207@gmail.com> References: <20110724164707.DB19A8208C@wyvern.cs.uni-duesseldorf.de> <4E2C5102.6090207@gmail.com> Message-ID: Hi Anto, On Sun, Jul 24, 2011 at 7:06 PM, Antonio Cuni wrote: > Note that pyrepl exists as an independent repo and that the one in PyPy is > supposed to be a copy of it (similar to what we do in py.test): > https://bitbucket.org/pypy/pyrepl Thanks for the reminder! I copied the diff in pypy/pyrepl with a checkin message that is a bit more verbose. A bient?t, Armin. From roberto at unbit.it Mon Jul 25 13:42:30 2011 From: roberto at unbit.it (Roberto De Ioris) Date: Mon, 25 Jul 2011 13:42:30 +0200 (CEST) Subject: [pypy-dev] work on libpypy.so Message-ID: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it> Hi everyone, my company has a bunch of C apps (mainly the uWSGI application server container and a scriptable DNS server) embedding CPython in their core. We would like to start using PyPy, so our need is having a workable libpypy.so implementation (we do not need it to be compatible with CPython api, we can rewrite those parts, expecially in uWSGI, without problems). I have searched for some work already done, but it looks like no one is pushing it. I apply myself for working on this area (if no-one is already doing it). So if there are no objections, i would start posting my proposals (and code) in the next few days. -- Roberto De Ioris http://unbit.it From arigo at tunes.org Mon Jul 25 14:31:59 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Jul 2011 14:31:59 +0200 Subject: [pypy-dev] work on libpypy.so In-Reply-To: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it> References: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it> Message-ID: Hi Roberto, On Mon, Jul 25, 2011 at 1:42 PM, Roberto De Ioris wrote: > We would like to start using PyPy, so our need is having a workable > libpypy.so implementation (we do not need it to be compatible with CPython > api, we can rewrite those parts, expecially in uWSGI, without problems). Indeed, there is nothing like that so far. We need to think out the kind of C-level API which makes sense for libpypy.so to expose. There are two options: either we tweak a little bit cpyext to expose the CPython C API, or we design some higher-performance PyPy-only C API... For the latter case the first issue is how to store a reference to a PyPy object in C, given that it moves around --- maybe with an indirection, which would be a bit similar to what cpyext does but can be done with much better performance if we are free to design the API. Feel free to come to our IRC channel to discuss things (#pypy on irc.freenode.net). A bient?t, Armin. From amauryfa at gmail.com Mon Jul 25 14:36:44 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 25 Jul 2011 14:36:44 +0200 Subject: [pypy-dev] work on libpypy.so In-Reply-To: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it> References: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it> Message-ID: Hi, 2011/7/25 Roberto De Ioris > > Hi everyone, my company has a bunch of C apps (mainly the uWSGI > application server container and a scriptable DNS server) embedding > CPython in their core. > > We would like to start using PyPy, so our need is having a workable > libpypy.so implementation (we do not need it to be compatible with CPython > api, we can rewrite those parts, expecially in uWSGI, without problems). > > I have searched for some work already done, but it looks like no one is > pushing it. I apply myself for working on this area (if no-one is already > doing it). So if there are no objections, i would start posting my > proposals (and code) in the next few days. Good idea! I suppose you already noticed the "--shared" option that already builds a libpypy-c.so. I've never tried it on Linux, but it works at least on Windows, and exposes all C API functions from cpyext. Still missing is the Py_Initialize() function... Could it be as simple as a single call to space.startup()? -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Mon Jul 25 16:40:36 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 25 Jul 2011 16:40:36 +0200 Subject: [pypy-dev] work on libpypy.so In-Reply-To: References: <49c7123ff5c207e38d3c0170e041223b.squirrel@manage.unbit.it> Message-ID: Armin Rigo, 25.07.2011 14:31: > On Mon, Jul 25, 2011 at 1:42 PM, Roberto De Ioris wrote: >> We would like to start using PyPy, so our need is having a workable >> libpypy.so implementation (we do not need it to be compatible with CPython >> api, we can rewrite those parts, expecially in uWSGI, without problems). > > Indeed, there is nothing like that so far. We need to think out the > kind of C-level API which makes sense for libpypy.so to expose. Wouldn't it make sense to use the Cython port for PyPy here? Usually, embedding just means that there is a two-way bridge between Python and C (or a C-compatible language). Cython basically gives you this, and allows you to design your C-level API any way you want. However, this means that it has to generate a C module that exports the public C names in one way or another. That won't work with the current approach, which only "compiles" down to Python code that uses ctypes (i.e. the other side). But it shouldn't be too hard to come up with a way to pass calls from the outside C-world into some kind of PyPy API that dispatches them to the appropriate Cython backed implementation. Basically, the external C code would only see a thin generated wrapper for each exported function (or even object) which maps the incoming parameters to ctypes objects, and the complete implementation of the user provided wrapper then runs in PyPy using ctypes. That would allow the PyPy core developers to design an appropriate public convenience C-API for PyPy, and at the same time allow users to design their own specific C-API for their own code, both using the same tool. Stefan From tobami at googlemail.com Mon Jul 25 21:09:34 2011 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 25 Jul 2011 21:09:34 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> Message-ID: Btw., in any case you can start saving results in separated environments. That will at least make changes work right away, and it does make sense to have "tannit 32 bits" and "tannit 64 bits". Miquel 2011/7/21 Miquel Torres : > Ok, not much time right now, but we will look into it. > > > > 2011/7/21 Maciej Fijalkowski : >> On Thu, Jul 21, 2011 at 1:52 PM, Antonio Cuni wrote: >>> On 21/07/11 10:02, Miquel Torres wrote: >>>> Alternatively, the timeline view could allow to display several >>>> environments at the same time... >>> >>> I think this would work. How hard is it to implement? >>> Maciek, any opinion? >> >> No, not much opinion from me >> >>> >>> ciao, >>> Anto >>> >> > From anto.cuni at gmail.com Tue Jul 26 14:09:54 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 26 Jul 2011 14:09:54 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> Message-ID: <4E2EAE92.8040905@gmail.com> On 25/07/11 21:09, Miquel Torres wrote: > Btw., in any case you can start saving results in separated > environments. That will at least make changes work right away, and it > does make sense to have "tannit 32 bits" and "tannit 64 bits". ok, I did it. If everything goes well, we should have two separate "tannit" and "tannit-64" environments from tomorrow. ciao, Anto From tobami at googlemail.com Tue Jul 26 18:41:06 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 26 Jul 2011 18:41:06 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: <4E2EAE92.8040905@gmail.com> References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> <4E2EAE92.8040905@gmail.com> Message-ID: Very nice! 2011/7/26 Antonio Cuni : > On 25/07/11 21:09, Miquel Torres wrote: >> >> Btw., in any case you can start saving results in separated >> environments. That will at least make changes work right away, and it >> does make sense to have "tannit 32 bits" and "tannit 64 bits". > > ok, I did it. ?If everything goes well, we should have two separate "tannit" > and "tannit-64" environments from tomorrow. > > ciao, > Anto > From anto.cuni at gmail.com Wed Jul 27 11:44:04 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 27 Jul 2011 11:44:04 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> Message-ID: <4E2FDDE4.3050109@gmail.com> Hi Miquel, On 25/07/11 21:09, Miquel Torres wrote: > Btw., in any case you can start saving results in separated > environments. That will at least make changes work right away, and it > does make sense to have "tannit 32 bits" and "tannit 64 bits". are you sure that having two separate environments will fix the "changes" page? I tried to add "tannit-64" (no results yet, a build is running right now), but in the dropdown menu of the Changes page I can see all the revisions that are also in "tannit". I'd have expected those two to be completely separate. ciao, Anto From fijall at gmail.com Tue Jul 26 09:51:08 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Jul 2011 09:51:08 +0200 Subject: [pypy-dev] [pypy-commit] pypy default: Remove some more importants, and reorganize some others. In-Reply-To: <20110726022505.DFD10829C2@wyvern.cs.uni-duesseldorf.de> References: <20110726022505.DFD10829C2@wyvern.cs.uni-duesseldorf.de> Message-ID: On Tue, Jul 26, 2011 at 4:25 AM, alex_gaynor wrote: > Author: Alex Gaynor > Branch: > Changeset: r45994:95649f1056c0 > Date: 2011-07-25 18:41 -0700 > http://bitbucket.org/pypy/pypy/changeset/95649f1056c0/ > > Log: ? ?Remove some more importants, and reorganize some others. > Please don't remove anything important :) From tobami at googlemail.com Thu Jul 28 11:04:56 2011 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 28 Jul 2011 11:04:56 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: <4E2FDDE4.3050109@gmail.com> References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> <4E2FDDE4.3050109@gmail.com> Message-ID: Hi Antonio, I'm afraid you are right, the solution I proposed makes no sense. Sorry I gave you a wrong answer. A revision is unique for a project (well, now to a branch of a project), and thus they are not separated by environment. Codespeed was not really designed with revisions in mind that sometimes have results, and sometimes not. To solve that, revisions would need to depend on an executable as well, or introduce a check that so that the revision list is tailored to a particular exe, but it would be ugly. There is a way though to "solve" it right now. Separate executables into two different projects. pypy32, pypy64, instead of different environements. The revision list shown does change on-the-fly depending on the project the selected exe belongs to. Cheers, Miquel 2011/7/27 Antonio Cuni : > Hi Miquel, > > On 25/07/11 21:09, Miquel Torres wrote: >> >> Btw., in any case you can start saving results in separated >> environments. That will at least make changes work right away, and it >> does make sense to have "tannit 32 bits" and "tannit 64 bits". > > are you sure that having two separate environments will fix the "changes" > page? I tried to add "tannit-64" (no results yet, a build is running right > now), but in the dropdown menu of the Changes page I can see all the > revisions that are also in "tannit". I'd have expected those two to be > completely separate. > > ciao, > Anto > From anto.cuni at gmail.com Thu Jul 28 12:27:21 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 28 Jul 2011 12:27:21 +0200 Subject: [pypy-dev] PyPy at last infinitely fast In-Reply-To: References: <4E248CCC.6090800@gmail.com> <4E2532FB.4030006@gmail.com> <4E27D09B.8040204@gmail.com> <4E27D447.3070807@gmail.com> <4E2812FE.8000407@gmail.com> <4E2FDDE4.3050109@gmail.com> Message-ID: <4E313989.3030907@gmail.com> Hi Miquel, On 28/07/11 11:04, Miquel Torres wrote: > Hi Antonio, > > I'm afraid you are right, the solution I proposed makes no sense. > Sorry I gave you a wrong answer. A revision is unique for a project > (well, now to a branch of a project), and thus they are not separated > by environment. Codespeed was not really designed with revisions in > mind that sometimes have results, and sometimes not. To solve that, > revisions would need to depend on an executable as well, or introduce > a check that so that the revision list is tailored to a particular > exe, but it would be ugly. I can think of a semi-ugly workaround, I don't know how close it is to a working solution. If I understand correctly, the numbers which are displayed in the "changes" page are precomputed and saved into "Reports" by create_report_if_enough_data. I can see that the function finds the last_results to compare with simply by doing last_revs[1]. What about changing this logic to pick the latest rev which actually contains at least one result for the current environment? > There is a way though to "solve" it right now. Separate executables > into two different projects. pypy32, pypy64, instead of different > environements. The revision list shown does change on-the-fly > depending on the project the selected exe belongs to. uhm, I don't see any option to select a different project from within speed.pypy.org. Is it simply because there is only one? Or having another project would mean to visit a complete different webpage? Moreover, I wonder how this problem relates to the upcoming speed.python.org: will every interpreter (cpython, pypy, jython, etc.) be a separate project? Will it be possible to compare results of different interpreters? ciao, Anto From fijall at gmail.com Thu Jul 28 18:27:05 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 28 Jul 2011 18:27:05 +0200 Subject: [pypy-dev] [pypy-commit] pypy default: don't raise a warning if the restype is not set, and add a test to check that we hit the fastpath even in that case In-Reply-To: <20110728141150.20CE882110@wyvern.cs.uni-duesseldorf.de> References: <20110728141150.20CE882110@wyvern.cs.uni-duesseldorf.de> Message-ID: > > Log: ? ?don't raise a warning if the restype is not set, and add a test to > ? ? ? ?check that we hit the fastpath even in that case Uh, can we revert this? I think a runtime warning when restype is not set is an excellent idea from the correctness perspective. From arigo at tunes.org Thu Jul 28 18:51:48 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Jul 2011 18:51:48 +0200 Subject: [pypy-dev] [pypy-commit] pypy default: don't raise a warning if the restype is not set, and add a test to check that we hit the fastpath even in that case In-Reply-To: References: <20110728141150.20CE882110@wyvern.cs.uni-duesseldorf.de> Message-ID: Hi Maciej, On Thu, Jul 28, 2011 at 6:27 PM, Maciej Fijalkowski wrote: > Uh, can we revert this? I think a runtime warning when restype is not > set is an excellent idea from the correctness perspective. I'm not sure I agree with this argument. The point I made on irc was that for function that return "void", it's common to not worry about setting restype to None, because it just works anyway. You get a c_long() object with a random value and just ignore it. It would be completely optimized away by the JIT, too. So yes, I see the point from a theoretical correctness perspective, but I think I care more about not flooding users with RuntimeWarnings in the case of ctypes... A bient?t, Armin. From Tom_Roche at pobox.com Sat Jul 30 18:54:29 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sat, 30 Jul 2011 12:54:29 -0400 Subject: [pypy-dev] downloads page, was: PyPy is now available in Fedora Message-ID: <87ei17g14q.fsf@pobox.com> David Malcolm Mon Jan 3 23:13:38 CET 2011 > I've packaged pypy in RPM form for the Fedora distribution and RPMs are available @, e.g., http://rpm.pbone.net/index.php3/stat/4/idpl/15770592/dir/rawhide/com/pypy-1.5-1.fc16.i686.rpm.html Shouldn't this be linked to, or at least mentioned, on http://pypy.org/download.html ? Feel free to forward if this is not the appropriate venue for this suggestion. fwiw, Tom Roche From fijall at gmail.com Sat Jul 30 19:16:57 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 30 Jul 2011 19:16:57 +0200 Subject: [pypy-dev] downloads page, was: PyPy is now available in Fedora In-Reply-To: <87ei17g14q.fsf@pobox.com> References: <87ei17g14q.fsf@pobox.com> Message-ID: On Sat, Jul 30, 2011 at 6:54 PM, Tom Roche wrote: > > David Malcolm Mon Jan 3 23:13:38 CET 2011 >> I've packaged pypy in RPM form for the Fedora distribution > > and RPMs are available @, e.g., > > http://rpm.pbone.net/index.php3/stat/4/idpl/15770592/dir/rawhide/com/pypy-1.5-1.fc16.i686.rpm.html > > Shouldn't this be linked to, or at least mentioned, on > > http://pypy.org/download.html > > ? Feel free to forward if this is not the appropriate venue for this suggestion. > It totally should (and here is a correct venue). I'll add it From arigo at tunes.org Sat Jul 30 23:50:22 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 30 Jul 2011 23:50:22 +0200 Subject: [pypy-dev] pypy array module memory leak? In-Reply-To: <4E2D1446.50606@ucs-software.co.za> References: <4E295417.9000903@ucs-software.co.za> <4E2D1446.50606@ucs-software.co.za> Message-ID: Hi Berend, I think I fixed the original problem too. See the longish checkin message of e7121092d73f. A bient?t, Armin. From Tom_Roche at pobox.com Sun Jul 31 00:17:34 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sat, 30 Jul 2011 18:17:34 -0400 Subject: [pypy-dev] newbie needs pypy setup help Message-ID: <874o23fm69.fsf@pobox.com> (Apologies if - this is not the appropriate venue for this question, but I don't see any other pypy lists, and I'm finding the other docs a bit inscrutable. - I sound too inexperienced or unpythonic: I've spent most of my coding life in java and perl. ) Please advise how to configure pypy to run other python code. Why I ask: I've started running a model implemented in python. Unfortunately a run on "normal" python 2.6.x or 2.7.x requires - 130 min on my ubuntu laptop (on which working would be more convenient) - 55 min on the best build machine to which I currently have access However I have read that this model runs 5x faster under pypy, so I wanna get me that! Unfortunately my current ubuntu me at it:~$ lsb_release -ds > Ubuntu 10.04.3 LTS # yes, I am planning to upgrade Real Soon Now me at it:~$ uname -rv > 2.6.32-33-generic #70-Ubuntu SMP Thu Jul 7 21:13:52 UTC 2011 is too down-level to install the available pypy RPM, so I instead did URI="https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2" TMP_DIR_ROOT="/tmp/pypy" for CMD in \ "rm -fr ${TMP_DIR_ROOT}" \ "mkdir -p ${TMP_DIR_ROOT}" \ "pushd ${TMP_DIR_ROOT}" \ "wget -O - ${URI} | tar xvjf -" \ "ls -alh" \ "popd" \ ; do echo -e "${CMD}" eval "${CMD}" done # check path names, then TMP_DIR_ROOT="/tmp/pypy/pypy-c-jit-43780-b590cf6de419-linux64" TARGET_DIR_ROOT="/opt/pypy-c-jit-1.5.0-alpha0" TARGET_PYPY_EXEC="${TARGET_DIR_ROOT}/bin/pypy" USR_PYPY_EXEC="/usr/local/bin/pypy" for CMD in \ "sudo mkdir -p ${TARGET_DIR_ROOT}" \ "sudo cp -r ${TMP_DIR_ROOT}/* ${TARGET_DIR_ROOT}/" \ "sudo chmod 755 ${TARGET_DIR_ROOT}/" \ "sudo chmod 755 ${TARGET_DIR_ROOT}/bin/" \ "sudo chmod 755 ${TARGET_PYPY_EXEC}" \ "sudo ls -al ${TARGET_PYPY_EXEC}" \ "sudo ln -s ${TARGET_PYPY_EXEC} ${USR_PYPY_EXEC}" \ "sudo ls -al ${USR_PYPY_EXEC}" \ "${USR_PYPY_EXEC} --version" \ "which pypy" \ "pypy --version" \ ; do echo -e "${CMD}" eval "${CMD}" done The good news is, I can now me at it:~$ ls -al $(which pypy) > lrwxrwxrwx 1 root root 37 2011-07-30 16:06 /usr/local/bin/pypy -> /opt/pypy-c-jit-1.5.0-alpha0/bin/pypy me at it:~$ pypy --version > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > [PyPy 1.5.0-alpha0 with GCC 4.4.3] But when I try to actually *run* the @#$%^&! thing, it spews: me at it:~$ pypy > debug: WARNING: library path not found, using compiled-in sys.path and sys.prefix will be unset > 'import site' failed > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: No module named _pypy_interact me at it:~$ pypy -c 'import sys; print sys.path' > debug: WARNING: library path not found, using compiled-in sys.path and sys.prefix will be unset > 'import site' failed > ['', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib_pypy', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7/lib-tk', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/lib-tk', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/plat-linux2'] What do I need to do to fix its library path? Is this where translation is required? (Again, please note that I *have read* http://codespeak.net/pypy/dist/pypy/doc/getting-started-python.html but I'm finding difficulty understanding what one must do to *use* pypy, vs what one must do to *hack* pypy, which is not my usecase.) TIA, Tom Roche From drsalists at gmail.com Sun Jul 31 01:15:08 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 30 Jul 2011 16:15:08 -0700 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <874o23fm69.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> Message-ID: Any particular reason not to use the tarball on the pypy website? There might be a silly permission issue to fix, but it's much easier than what appears below. On Sat, Jul 30, 2011 at 3:17 PM, Tom Roche wrote: > > (Apologies if > > - this is not the appropriate venue for this question, but I don't see > any other pypy lists, and I'm finding the other docs a bit > inscrutable. > > - I sound too inexperienced or unpythonic: I've spent most of my coding > life in java and perl. > > ) > Please advise how to configure pypy to run other python code. Why I ask: > > I've started running a model implemented in python. Unfortunately a run > on "normal" python 2.6.x or 2.7.x requires > > - 130 min on my ubuntu laptop (on which working would be more convenient) > - 55 min on the best build machine to which I currently have access > > However I have read that this model runs 5x faster under pypy, so I > wanna get me that! Unfortunately my current ubuntu > > me at it:~$ lsb_release -ds > > Ubuntu 10.04.3 LTS # yes, I am planning to upgrade Real Soon Now > me at it:~$ uname -rv > > 2.6.32-33-generic #70-Ubuntu SMP Thu Jul 7 21:13:52 UTC 2011 > > is too down-level to install the available pypy RPM, so I instead did > > URI="https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2" > TMP_DIR_ROOT="/tmp/pypy" > for CMD in \ > "rm -fr ${TMP_DIR_ROOT}" \ > "mkdir -p ${TMP_DIR_ROOT}" \ > "pushd ${TMP_DIR_ROOT}" \ > "wget -O - ${URI} | tar xvjf -" \ > "ls -alh" \ > "popd" \ > ; do > echo -e "${CMD}" > eval "${CMD}" > done > # check path names, then > TMP_DIR_ROOT="/tmp/pypy/pypy-c-jit-43780-b590cf6de419-linux64" > TARGET_DIR_ROOT="/opt/pypy-c-jit-1.5.0-alpha0" > TARGET_PYPY_EXEC="${TARGET_DIR_ROOT}/bin/pypy" > USR_PYPY_EXEC="/usr/local/bin/pypy" > for CMD in \ > "sudo mkdir -p ${TARGET_DIR_ROOT}" \ > "sudo cp -r ${TMP_DIR_ROOT}/* ${TARGET_DIR_ROOT}/" \ > "sudo chmod 755 ${TARGET_DIR_ROOT}/" \ > "sudo chmod 755 ${TARGET_DIR_ROOT}/bin/" \ > "sudo chmod 755 ${TARGET_PYPY_EXEC}" \ > "sudo ls -al ${TARGET_PYPY_EXEC}" \ > "sudo ln -s ${TARGET_PYPY_EXEC} ${USR_PYPY_EXEC}" \ > "sudo ls -al ${USR_PYPY_EXEC}" \ > "${USR_PYPY_EXEC} --version" \ > "which pypy" \ > "pypy --version" \ > ; do > echo -e "${CMD}" > eval "${CMD}" > done > > The good news is, I can now > > me at it:~$ ls -al $(which pypy) > > lrwxrwxrwx 1 root root 37 2011-07-30 16:06 /usr/local/bin/pypy -> > /opt/pypy-c-jit-1.5.0-alpha0/bin/pypy > me at it:~$ pypy --version > > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > > [PyPy 1.5.0-alpha0 with GCC 4.4.3] > > But when I try to actually *run* the @#$%^&! thing, it spews: > > me at it:~$ pypy > > debug: WARNING: library path not found, using compiled-in sys.path and > sys.prefix will be unset > > 'import site' failed > > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > > [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > debug: OperationError: > > debug: operror-type: ImportError > > debug: operror-value: No module named _pypy_interact > me at it:~$ pypy -c 'import sys; print sys.path' > > debug: WARNING: library path not found, using compiled-in sys.path and > sys.prefix will be unset > > 'import site' failed > > ['', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib_pypy', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7/lib-tk', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/lib-tk', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/plat-linux2'] > > What do I need to do to fix its library path? Is this where translation > is required? (Again, please note that I *have read* > > http://codespeak.net/pypy/dist/pypy/doc/getting-started-python.html > > but I'm finding difficulty understanding what one must do to *use* pypy, > vs what one must do to *hack* pypy, which is not my usecase.) > > TIA, Tom Roche > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Sun Jul 31 01:31:47 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 01:31:47 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <874o23fm69.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> Message-ID: [snip] It's about pypy not finding it's library. I have a bit no idea how those RPMs are built, but usually putting pypy in the same directory tree as it's library (like in checkout) works. I agree we should maybe make some document that makes it more explicit From Tom_Roche at pobox.com Sun Jul 31 02:32:50 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sat, 30 Jul 2011 20:32:50 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> Message-ID: <871ux7ffwt.fsf@pobox.com> Tom Roche Sat, 30 Jul 2011 18:17:34 -0400 >>> my current ubuntu >>> me at it:~$ lsb_release -ds >>> > Ubuntu 10.04.3 LTS # yes, I am planning to upgrade Real Soon Now >>> me at it:~$ uname -rv >>> > 2.6.32-33-generic #70-Ubuntu SMP Thu Jul 7 21:13:52 UTC 2011 >>> is too down-level to install the available pypy RPM, so I instead >>> [downloaded and installed] >>> URI="https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2" Dan Stromberg Sat, 30 Jul 2011 16:15:08 -0700 >> Any particular reason not to use the tarball on the pypy website? That is in fact the "Linux binary (64bit)" from http://pypy.org/download.html#default-with-a-jit-compiler Maciej Fijalkowski Sun, 31 Jul 2011 01:31:47 +0200 > I have a bit no idea how those RPMs are built, Again, please note: I could not use the RPM, so instead used the tarball. Your assistance is appreciated, Tom Roche From Tom_Roche at pobox.com Sun Jul 31 03:58:36 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sat, 30 Jul 2011 21:58:36 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <871ux7ffwt.fsf@pobox.com> References: <871ux7ffwt.fsf@pobox.com> <874o23fm69.fsf@pobox.com> Message-ID: <87y5zfdxdf.fsf@pobox.com> Dan Stromberg Sat, 30 Jul 2011 16:15:08 -0700 > There might be a silly permission issue Or more than one. I noticed me at it:~$ pushd ${TARGET_DIR_ROOT} # i.e. where I'm copying pypy to, in /opt me at it:/opt/pypy-c-jit-1.5.0-alpha0$ ls -alh > total 440K > drwxr-xr-x 8 root root 4.0K 2011-07-30 17:00 . > drwxr-xr-x 8 root root 4.0K 2011-07-30 16:06 .. > drwxr-xr-x 2 root root 4.0K 2011-07-30 16:45 bin > drwxr-xr-x 2 root root 4.0K 2011-07-30 17:00 build > drwx------ 2 root root 4.0K 2011-07-30 16:06 include > drwx------ 7 root root 4.0K 2011-07-30 16:44 lib_pypy > drwx------ 4 root root 4.0K 2011-07-30 16:06 lib-python > -rw------- 1 root root 5.8K 2011-07-30 16:06 LICENSE > -rw-r--r-- 1 root root 748 2011-07-30 16:06 README > drwx------ 5 root root 4.0K 2011-07-30 16:45 site-packages which seems wrong (even to me :-) So I whacked that install (below) and rebuilt with the following: please note * the chmod's * again, that I'm using the "Linux binary (64bit)" tarball (as previously) URI="https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2" TMP_DIR_ROOT="/tmp/pypy/pypy-c-jit-43780-b590cf6de419-linux64" TARGET_DIR_ROOT="/opt/pypy-c-jit-1.5.0-alpha0" TARGET_PYPY_EXEC="${TARGET_DIR_ROOT}/bin/pypy" USR_PYPY_EXEC="/usr/local/bin/pypy" for CMD in \ "rm -fr $(dirname ${TMP_DIR_ROOT})" \ "mkdir -p ${TMP_DIR_ROOT}" \ "pushd $(dirname ${TMP_DIR_ROOT})" \ "wget -O - ${URI} | tar xvjf -" \ "ls -alh" \ "popd" \ "sudo rm -fr ${TARGET_DIR_ROOT}" \ "sudo mkdir -p ${TARGET_DIR_ROOT}" \ "sudo cp -r ${TMP_DIR_ROOT}/* ${TARGET_DIR_ROOT}/" \ "sudo chmod a+rx ${TARGET_DIR_ROOT}/" \ "pushd ${TARGET_DIR_ROOT}" \ "find -maxdepth 1 -type f | xargs sudo chmod a+r" \ "find -maxdepth 1 -type d | grep -ve '\.$' | xargs sudo chmod a+rx" \ "sudo chmod a+rx ${TARGET_PYPY_EXEC}" \ "sudo ls -al ${TARGET_PYPY_EXEC}" \ "sudo ln -s ${TARGET_PYPY_EXEC} ${USR_PYPY_EXEC}" \ "which pypy" \ "sudo ls -al ${USR_PYPY_EXEC}" \ "${USR_PYPY_EXEC} --version" \ "sudo ls -al ${TARGET_DIR_ROOT}" \ "pypy --version" \ "pypy" \ "pypy -c 'import sys; print sys.path'" \ "popd" \ ; do echo -e "$ ${CMD}" eval "${CMD}" done The tail of the resulting output is > $ sudo ls -al /opt/pypy-c-jit-1.5.0-alpha0 > total 40 > drwxr-xr-x 7 root root 4096 2011-07-30 21:44 . > drwxr-xr-x 8 root root 4096 2011-07-30 21:44 .. > drwxr-xr-x 2 root root 4096 2011-07-30 21:44 bin > drwxr-xr-x 2 root root 4096 2011-07-30 21:44 include > drwxr-xr-x 7 root root 4096 2011-07-30 21:44 lib_pypy > drwxr-xr-x 4 root root 4096 2011-07-30 21:44 lib-python > -rw-r--r-- 1 root root 5921 2011-07-30 21:44 LICENSE > -rw-r--r-- 1 root root 748 2011-07-30 21:44 README > drwxr-xr-x 2 root root 4096 2011-07-30 21:44 site-packages Those permissions look better than above/before, and > $ pypy --version > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > [PyPy 1.5.0-alpha0 with GCC 4.4.3] still looks good, but > $ pypy > 'import site' failed > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: No module named _pypy_interact is still hosed! though at least pypy is now finding its library path: > $ pypy -c 'import sys; print sys.path' > 'import site' failed > ['', '/opt/pypy-c-jit-1.5.0-alpha0/lib_pypy', '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/modified-2.7', '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/2.7', '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/modified-2.7/lib-tk', '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/2.7/lib-tk', '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/2.7/plat-linux2'] Which it wasn't before: Tom Roche Sat, 30 Jul 2011 18:17:34 -0400 >> me at it:~$ pypy -c 'import sys; print sys.path' >> > debug: WARNING: library path not found, using compiled-in sys.path and sys.prefix will be unset >> > 'import site' failed >> > ['', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib_pypy', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7/lib-tk', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/lib-tk', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/plat-linux2'] So I'm wondering, what else needs done in order to make this work? E.g., how to fix the "'import site' failed"? TIA, Tom Roche From Tom_Roche at pobox.com Sun Jul 31 04:14:54 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sat, 30 Jul 2011 22:14:54 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <87y5zfdxdf.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <874o23fm69.fsf@pobox.com> Message-ID: <87vcujdwm9.fsf@pobox.com> Ken Watford Sat, 30 Jul 2011 21:18:23 -0400 > sudo chmod -R go+rX /opt/pypy-c-jit-1.5.0-alpha0 That's it! So the tarball install that works (for me) is URI="https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2" TMP_DIR_ROOT="/tmp/pypy/pypy-c-jit-43780-b590cf6de419-linux64" TARGET_DIR_ROOT="/opt/pypy-c-jit-1.5.0-alpha0" TARGET_PYPY_EXEC="${TARGET_DIR_ROOT}/bin/pypy" USR_PYPY_EXEC="/usr/local/bin/pypy" for CMD in \ "rm -fr $(dirname ${TMP_DIR_ROOT})" \ "mkdir -p ${TMP_DIR_ROOT}" \ "pushd $(dirname ${TMP_DIR_ROOT})" \ "wget -O - ${URI} | tar xvjf -" \ "ls -alh" \ "popd" \ "sudo rm -fr ${TARGET_DIR_ROOT}" \ "sudo mkdir -p ${TARGET_DIR_ROOT}" \ "sudo cp -r ${TMP_DIR_ROOT}/* ${TARGET_DIR_ROOT}/" \ "sudo chmod -R go+rX ${TARGET_DIR_ROOT}/" \ "sudo ln -s ${TARGET_PYPY_EXEC} ${USR_PYPY_EXEC}" \ "which pypy" \ "sudo ls -al ${USR_PYPY_EXEC}" \ "${USR_PYPY_EXEC} --version" \ "sudo ls -al ${TARGET_DIR_ROOT}" \ "pypy --version" \ "pypy" \ "popd" \ ; do echo -e "$ ${CMD}" eval "${CMD}" done which tails > $ pypy > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > And now for something completely different: ``the future has just begun'' > >>>> Thanks! Tom Roche From drsalists at gmail.com Sun Jul 31 04:15:52 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 30 Jul 2011 19:15:52 -0700 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <87y5zfdxdf.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <87y5zfdxdf.fsf@pobox.com> Message-ID: I usually: 1) chown it all to root. 2) chmod all the directories and bin/pypy to 755 3) chmod all the regular files except bin/pypy to 644 ...and then I end up with a usable pypy. Or I build it myself - that doesn't give weird permissions problems. After building it, I've been using this script to do the install: #!/bin/bash #where=/usr/local/pypy-trunk-2010-11-20 if [ "$where" = "" ] then echo Need a '$where' to be a prefix path 1>&2 exit 1 fi if [ -d "$where" ] then echo "$where already exists" 1>&2 exit 1 fi # This is for building it: # cd pypy/translator/goal # python translate.py --opt=jit targetpypystandalone.py if [ -f "pypy-c" ] then : else echo "Sorry, I don't see a pypy-c in the CWD" exit 1 fi export where mkdir -p "$where"/bin cp pypy-c "$where"/bin/pypy cd ../../.. mkdir "$where"/include cp include/* "$where"/include/. mkdir "$where"/lib-python (cd lib-python/ && tar cflS - .) | (cd "$where"/lib-python && tar xfp -) mkdir -p "$where"/lib_pypy (cd lib_pypy && tar cflS - .) | (cd "$where"/lib_pypy && tar xfp -) #(cd translator/sandbox && tar cflS - .) | (cd "$where"/lib_pypy && tar xfp -) find "$where" -name .svn -print | xargs rm -rf On Sat, Jul 30, 2011 at 6:58 PM, Tom Roche wrote: > > Dan Stromberg Sat, 30 Jul 2011 16:15:08 -0700 > > There might be a silly permission issue > > Or more than one. I noticed > > me at it:~$ pushd ${TARGET_DIR_ROOT} # i.e. where I'm copying pypy to, in > /opt > me at it:/opt/pypy-c-jit-1.5.0-alpha0$ ls -alh > > total 440K > > drwxr-xr-x 8 root root 4.0K 2011-07-30 17:00 . > > drwxr-xr-x 8 root root 4.0K 2011-07-30 16:06 .. > > drwxr-xr-x 2 root root 4.0K 2011-07-30 16:45 bin > > drwxr-xr-x 2 root root 4.0K 2011-07-30 17:00 build > > drwx------ 2 root root 4.0K 2011-07-30 16:06 include > > drwx------ 7 root root 4.0K 2011-07-30 16:44 lib_pypy > > drwx------ 4 root root 4.0K 2011-07-30 16:06 lib-python > > -rw------- 1 root root 5.8K 2011-07-30 16:06 LICENSE > > -rw-r--r-- 1 root root 748 2011-07-30 16:06 README > > drwx------ 5 root root 4.0K 2011-07-30 16:45 site-packages > > which seems wrong (even to me :-) So I whacked that install (below) and > rebuilt with the following: please note > > * the chmod's > > * again, that I'm using the "Linux binary (64bit)" tarball (as previously) > > URI="https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2" > TMP_DIR_ROOT="/tmp/pypy/pypy-c-jit-43780-b590cf6de419-linux64" > TARGET_DIR_ROOT="/opt/pypy-c-jit-1.5.0-alpha0" > TARGET_PYPY_EXEC="${TARGET_DIR_ROOT}/bin/pypy" > USR_PYPY_EXEC="/usr/local/bin/pypy" > for CMD in \ > "rm -fr $(dirname ${TMP_DIR_ROOT})" \ > "mkdir -p ${TMP_DIR_ROOT}" \ > "pushd $(dirname ${TMP_DIR_ROOT})" \ > "wget -O - ${URI} | tar xvjf -" \ > "ls -alh" \ > "popd" \ > "sudo rm -fr ${TARGET_DIR_ROOT}" \ > "sudo mkdir -p ${TARGET_DIR_ROOT}" \ > "sudo cp -r ${TMP_DIR_ROOT}/* ${TARGET_DIR_ROOT}/" \ > "sudo chmod a+rx ${TARGET_DIR_ROOT}/" \ > "pushd ${TARGET_DIR_ROOT}" \ > "find -maxdepth 1 -type f | xargs sudo chmod a+r" \ > "find -maxdepth 1 -type d | grep -ve '\.$' | xargs sudo chmod a+rx" \ > "sudo chmod a+rx ${TARGET_PYPY_EXEC}" \ > "sudo ls -al ${TARGET_PYPY_EXEC}" \ > "sudo ln -s ${TARGET_PYPY_EXEC} ${USR_PYPY_EXEC}" \ > "which pypy" \ > "sudo ls -al ${USR_PYPY_EXEC}" \ > "${USR_PYPY_EXEC} --version" \ > "sudo ls -al ${TARGET_DIR_ROOT}" \ > "pypy --version" \ > "pypy" \ > "pypy -c 'import sys; print sys.path'" \ > "popd" \ > ; do > echo -e "$ ${CMD}" > eval "${CMD}" > done > > The tail of the resulting output is > > > $ sudo ls -al /opt/pypy-c-jit-1.5.0-alpha0 > > total 40 > > drwxr-xr-x 7 root root 4096 2011-07-30 21:44 . > > drwxr-xr-x 8 root root 4096 2011-07-30 21:44 .. > > drwxr-xr-x 2 root root 4096 2011-07-30 21:44 bin > > drwxr-xr-x 2 root root 4096 2011-07-30 21:44 include > > drwxr-xr-x 7 root root 4096 2011-07-30 21:44 lib_pypy > > drwxr-xr-x 4 root root 4096 2011-07-30 21:44 lib-python > > -rw-r--r-- 1 root root 5921 2011-07-30 21:44 LICENSE > > -rw-r--r-- 1 root root 748 2011-07-30 21:44 README > > drwxr-xr-x 2 root root 4096 2011-07-30 21:44 site-packages > > Those permissions look better than above/before, and > > > $ pypy --version > > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > > [PyPy 1.5.0-alpha0 with GCC 4.4.3] > > still looks good, but > > > $ pypy > > 'import site' failed > > Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) > > [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > debug: OperationError: > > debug: operror-type: ImportError > > debug: operror-value: No module named _pypy_interact > > is still hosed! though at least pypy is now finding its library path: > > > $ pypy -c 'import sys; print sys.path' > > 'import site' failed > > ['', '/opt/pypy-c-jit-1.5.0-alpha0/lib_pypy', > '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/modified-2.7', > '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/2.7', > '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/modified-2.7/lib-tk', > '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/2.7/lib-tk', > '/opt/pypy-c-jit-1.5.0-alpha0/lib-python/2.7/plat-linux2'] > > Which it wasn't before: > > Tom Roche Sat, 30 Jul 2011 18:17:34 -0400 > >> me at it:~$ pypy -c 'import sys; print sys.path' > >> > debug: WARNING: library path not found, using compiled-in sys.path and > sys.prefix will be unset > >> > 'import site' failed > >> > ['', '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib_pypy', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/modified-2.7/lib-tk', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/lib-tk', > '/home/buildslave/bot64/pypy-c-jit-linux-x86-64/build/lib-python/2.7/plat-linux2'] > > So I'm wondering, what else needs done in order to make this work? E.g., > how to fix the "'import site' failed"? > > TIA, Tom Roche > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Sun Jul 31 04:27:49 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 31 Jul 2011 02:27:49 -0000 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <871ux7ffwt.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> Message-ID: <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> On 12:32 am, tom_roche at pobox.com wrote: > >Tom Roche Sat, 30 Jul 2011 18:17:34 -0400 >>>>my current ubuntu > >>>>me at it:~$ lsb_release -ds >>>> > Ubuntu 10.04.3 LTS # yes, I am planning to upgrade Real Soon Now >>>>me at it:~$ uname -rv >>>> > 2.6.32-33-generic #70-Ubuntu SMP Thu Jul 7 21:13:52 UTC 2011 > >>>>is too down-level to install the available pypy RPM, so I instead >>>>[downloaded and installed] > >>>>URI="https://bitbucket.org/pypy/pypy/downloads/pypy-1.5-linux64.tar.bz2" > >Dan Stromberg Sat, 30 Jul 2011 16:15:08 -0700 >>>Any particular reason not to use the tarball on the pypy website? > >That is in fact the "Linux binary (64bit)" from >http://pypy.org/download.html#default-with-a-jit-compiler > >Maciej Fijalkowski Sun, 31 Jul 2011 01:31:47 +0200 >>I have a bit no idea how those RPMs are built, > >Again, please note: I could not use the RPM, so instead used the >tarball. Just unpack the tarball. Don't do all that copying of stuff. Jean-Paul From drsalists at gmail.com Sun Jul 31 05:09:58 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 30 Jul 2011 20:09:58 -0700 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> Message-ID: On Sat, Jul 30, 2011 at 7:27 PM, wrote: > Just unpack the tarball. Don't do all that copying of stuff. > > Jean-Paul > > I believe it's more than a matter of just unpacking a tar archive. The tar archives have had permissions problems long enough that I suspect there's a bug in the build/release process. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Sun Jul 31 19:48:26 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 19:48:26 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> Message-ID: On Sun, Jul 31, 2011 at 5:09 AM, Dan Stromberg wrote: > > On Sat, Jul 30, 2011 at 7:27 PM, wrote: >> >> Just unpack the tarball. ?Don't do all that copying of stuff. >> >> Jean-Paul > > I believe it's more than a matter of just unpacking a tar archive.? The tar > archives have had permissions problems long enough that I suspect there's a > bug in the build/release process. Precisely what didn't work? I run nightlies just fine by downloading and running them From kwatford at gmail.com Sun Jul 31 19:59:55 2011 From: kwatford at gmail.com (Ken Watford) Date: Sun, 31 Jul 2011 13:59:55 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> Message-ID: > Precisely what didn't work? I run nightlies just fine by downloading > and running them The permissions are fine if you're the owner of the files. If another user (say, root) untars them, other users may not have appropriate read/execute permissions. From Tom_Roche at pobox.com Sun Jul 31 21:03:34 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sun, 31 Jul 2011 15:03:34 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> Message-ID: <87mxfue0hl.fsf@pobox.com> Maciej Fijalkowski Sun, 31 Jul 2011 19:48:26 +0200 >> Precisely what didn't work? Ken Watford Sun Jul 31 19:59:55 CEST 2011 > The permissions are fine if you're the owner of the files. If another > user (say, root) untars them, e.g., to install FHS-ly > other users may not have appropriate read/execute permissions. FWIW, Tom Roche From fijall at gmail.com Sun Jul 31 21:07:17 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 21:07:17 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <87mxfue0hl.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> Message-ID: On Sun, Jul 31, 2011 at 9:03 PM, Tom Roche wrote: > > Maciej Fijalkowski Sun, 31 Jul 2011 19:48:26 +0200 >>> Precisely what didn't work? > > Ken Watford Sun Jul 31 19:59:55 CEST 2011 >> The permissions are fine if you're the owner of the files. If another >> user (say, root) untars them, > > e.g., to install FHS-ly Ok, as far as I understood, the tarballs are for extracing in-place not for installs without serious massaging > >> other users may not have appropriate read/execute permissions. > > FWIW, Tom Roche > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From fijall at gmail.com Sun Jul 31 21:07:39 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 21:07:39 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> Message-ID: On Sun, Jul 31, 2011 at 9:07 PM, Maciej Fijalkowski wrote: > On Sun, Jul 31, 2011 at 9:03 PM, Tom Roche wrote: >> >> Maciej Fijalkowski Sun, 31 Jul 2011 19:48:26 +0200 >>>> Precisely what didn't work? >> >> Ken Watford Sun Jul 31 19:59:55 CEST 2011 >>> The permissions are fine if you're the owner of the files. If another >>> user (say, root) untars them, >> >> e.g., to install FHS-ly > > Ok, as far as I understood, the tarballs are for extracing in-place > not for installs without serious massaging > Ah and btw, pypy used to be in debian but got removed From Tom_Roche at pobox.com Sun Jul 31 21:28:51 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sun, 31 Jul 2011 15:28:51 -0400 Subject: [pypy-dev] pypy for debian? Message-ID: <87k4aydzbg.fsf@pobox.com> Anyone packaging pypy for debian? Why I ask: I'm putting a FOSS project on ShiningPanda, which runs Debian 6 (squeeze), for CI. After seeing the speedup from pypy on my own hardware, I'd like to use pypy on SP, but they don't currently offer that option. Hence I'd like to ask SP to provide pypy along with the CPython versions they provide (rather than hafta provide it myself in each build environment I use, which would add to my setup time, defeating the purpose), but before I did that, I'd like to get some idea of how much work would be entailed, i.e., how much pain I'd be asking them to endure. Adopting pypy would presumably be trivial for SP if there were pypy .deb, or even a PPA, but I'm not seeing anything like that after ubuntu hardy. Am I missing something? I see RH (appears to be) regularly packaging pypy, so I'm hoping someone's doing this for debian. If not, there's always `alien`. TIA, Tom Roche From Tom_Roche at pobox.com Sun Jul 31 21:44:55 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sun, 31 Jul 2011 15:44:55 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> Message-ID: <87hb62dyko.fsf@pobox.com> Maciej Fijalkowski Sun, 31 Jul 2011 19:48:26 +0200 >>>> Precisely what didn't work? Ken Watford Sun Jul 31 19:59:55 CEST 2011 >>> The permissions are fine if you're the owner of the files. If another >>> user (say, root) untars them, Tom Roche Sun, 31 Jul 2011 15:03:34 -0400 >> e.g., to install FHS-ly >>> other users may not have appropriate read/execute permissions. Maciej Fijalkowski Sun, 31 Jul 2011 21:07:17 +0200 > as far as I understood, the tarballs are for [extracting] in-place not > for installs without serious massaging Not so much massage is required (e.g., my bash script from http://mail.python.org/pipermail/pypy-dev/2011-July/007893.html does it, as does Dan Stromberg's @ http://mail.python.org/pipermail/pypy-dev/2011-July/007894.html ) but It Would Be Nice for new adopters if the download page documented things like this, esp if intentional. Currently it says only http://pypy.org/download.html#installing > Installing > All versions are packaged in a tar.bz2 or zip file. When uncompressed, > they run in-place. For now you can uncompress them either somewhere in > your home directory or, say, in /opt, and if you want, put a symlink > from somewhere like /usr/local/bin/pypy to /path/to/pypy-1.5/bin/pypy. > Do not move or copy the executable pypy outside the tree ? put a > symlink to it, otherwise it will not find its libraries. FWIW, Tom Roche From fijall at gmail.com Sun Jul 31 21:47:16 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 21:47:16 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <87hb62dyko.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> Message-ID: On Sun, Jul 31, 2011 at 9:44 PM, Tom Roche wrote: > > Maciej Fijalkowski Sun, 31 Jul 2011 19:48:26 +0200 >>>>> Precisely what didn't work? > > Ken Watford Sun Jul 31 19:59:55 CEST 2011 >>>> The permissions are fine if you're the owner of the files. If another >>>> user (say, root) untars them, > > Tom Roche Sun, 31 Jul 2011 15:03:34 -0400 >>> e.g., to install FHS-ly > >>>> other users may not have appropriate read/execute permissions. > > Maciej Fijalkowski Sun, 31 Jul 2011 21:07:17 +0200 >> as far as I understood, the tarballs are for [extracting] in-place not >> for installs without serious massaging > > Not so much massage is required (e.g., my bash script from > > http://mail.python.org/pipermail/pypy-dev/2011-July/007893.html > > does it, as does Dan Stromberg's @ > > http://mail.python.org/pipermail/pypy-dev/2011-July/007894.html > > ) but It Would Be Nice for new adopters if the download page documented things like this, esp if intentional. Currently it says only > > http://pypy.org/download.html#installing >> Installing > >> All versions are packaged in a tar.bz2 or zip file. When uncompressed, >> they run in-place. For now you can uncompress them either somewhere in >> your home directory or, say, in /opt, and if you want, put a symlink >> from somewhere like /usr/local/bin/pypy to /path/to/pypy-1.5/bin/pypy. >> Do not move or copy the executable pypy outside the tree ? put a >> symlink to it, otherwise it will not find its libraries. > > FWIW, Tom Roche > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Cool, feel like providing a patch? Cheers, fijal From fijall at gmail.com Sun Jul 31 21:47:48 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 21:47:48 +0200 Subject: [pypy-dev] pypy for debian? In-Reply-To: <87k4aydzbg.fsf@pobox.com> References: <87k4aydzbg.fsf@pobox.com> Message-ID: On Sun, Jul 31, 2011 at 9:28 PM, Tom Roche wrote: > > Anyone packaging pypy for debian? Why I ask: > > I'm putting a FOSS project on ShiningPanda, which runs Debian 6 (squeeze), for CI. After seeing the speedup from pypy on my own hardware, I'd like to use pypy on SP, but they don't currently offer that option. Hence I'd like to ask SP to provide pypy along with the CPython versions they provide (rather than hafta provide it myself in each build environment I use, which would add to my setup time, defeating the purpose), but before I did that, I'd like to get some idea of how much work would be entailed, i.e., how much pain I'd be asking them to endure. > > Adopting pypy would presumably be trivial for SP if there were pypy .deb, or even a PPA, but I'm not seeing anything like that after ubuntu hardy. Am I missing something? I see RH (appears to be) regularly packaging pypy, so I'm hoping someone's doing this for debian. If not, there's always `alien`. I think what's actually needed is a PPA maintainer (or a debian champion). Cheers, fijal From Tom_Roche at pobox.com Sun Jul 31 22:03:12 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sun, 31 Jul 2011 16:03:12 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> Message-ID: <87ei16dxq7.fsf@pobox.com> Maciej Fijalkowski Sun, 31 Jul 2011 21:07:17 +0200 >>> as far as I understood, the tarballs are for [extracting] in-place not >>> for installs without serious massaging Tom Roche Sun, 31 Jul 2011 15:44:55 -0400 >> Not so much massage is required (e.g., my bash script from >> http://mail.python.org/pipermail/pypy-dev/2011-July/007893.html >> does it, as does Dan Stromberg's @ >> http://mail.python.org/pipermail/pypy-dev/2011-July/007894.html >> ) but It Would Be Nice for new adopters if the download page >> documented things like this, esp if intentional. Maciej Fijalkowski Sun, 31 Jul 2011 21:47:16 +0200 > Cool, feel like providing a patch? To the build or the doc? Regarding the latter, have youse thought about a wiki, or other vehicle for user-contributed doc? Minimally, you could point to http://wiki.python.org/moin/PyPy and have folks work there. Currently it says http://wiki.python.org/moin/PyPy > Please refer to the PyPy home page and development site (which appear to be the same page) > for more information about this Python implementation. which (IMHO, YMMV) kinda discourages putting more information on that wiki. FWIW, Tom Roche From Tom_Roche at pobox.com Sun Jul 31 22:03:57 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sun, 31 Jul 2011 16:03:57 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> Message-ID: <87bowadxoy.fsf@pobox.com> Maciej Fijalkowski Sun, 31 Jul 2011 21:07:17 +0200 >>> as far as I understood, the tarballs are for [extracting] in-place not >>> for installs without serious massaging Tom Roche Sun, 31 Jul 2011 15:44:55 -0400 >> Not so much massage is required (e.g., my bash script from >> http://mail.python.org/pipermail/pypy-dev/2011-July/007893.html >> does it, as does Dan Stromberg's @ >> http://mail.python.org/pipermail/pypy-dev/2011-July/007894.html >> ) but It Would Be Nice for new adopters if the download page >> documented things like this, esp if intentional. Maciej Fijalkowski Sun, 31 Jul 2011 21:47:16 +0200 > Cool, feel like providing a patch? To the build or the doc? Regarding the latter, have youse thought about a wiki, or other vehicle for user-contributed doc? Minimally, you could point to http://wiki.python.org/moin/PyPy and have folks work there. Currently it says http://wiki.python.org/moin/PyPy > Please refer to the PyPy home page and development site (which appear to be the same page) > for more information about this Python implementation. which (IMHO, YMMV) kinda discourages putting more information on that wiki. FWIW, Tom Roche From fijall at gmail.com Sun Jul 31 22:06:36 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 22:06:36 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <87ei16dxq7.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> <87ei16dxq7.fsf@pobox.com> Message-ID: On Sun, Jul 31, 2011 at 10:03 PM, Tom Roche wrote: > > Maciej Fijalkowski Sun, 31 Jul 2011 21:07:17 +0200 >>>> as far as I understood, the tarballs are for [extracting] in-place not >>>> for installs without serious massaging > > Tom Roche Sun, 31 Jul 2011 15:44:55 -0400 > >>> Not so much massage is required (e.g., my bash script from > >>> http://mail.python.org/pipermail/pypy-dev/2011-July/007893.html > >>> does it, as does Dan Stromberg's @ > >>> http://mail.python.org/pipermail/pypy-dev/2011-July/007894.html > >>> ) but It Would Be Nice for new adopters if the download page >>> documented things like this, esp if intentional. > > Maciej Fijalkowski Sun, 31 Jul 2011 21:47:16 +0200 >> Cool, feel like providing a patch? > > To the build or the doc? To the doc. We have a wiki on bitbucket (apparently underused) > > Regarding the latter, have youse thought about a wiki, or other vehicle for user-contributed doc? Minimally, you could point to > > http://wiki.python.org/moin/PyPy > > and have folks work there. Currently it says > > http://wiki.python.org/moin/PyPy >> Please refer to the PyPy home page and development site > > (which appear to be the same page) > >> for more information about this Python implementation. > > which (IMHO, YMMV) kinda discourages putting more information on that wiki. > > FWIW, Tom Roche > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From janzert at janzert.com Sun Jul 31 22:18:32 2011 From: janzert at janzert.com (Janzert) Date: Sun, 31 Jul 2011 16:18:32 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <87hb62dyko.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> Message-ID: Since I haven't seen it mentioned yet. The issue with permissions is supposed to already be fixed in trunk - https://bugs.pypy.org/issue772 so the next release should resolve that problem for system wide installs. Janzert From Tom_Roche at pobox.com Sun Jul 31 23:05:10 2011 From: Tom_Roche at pobox.com (Tom Roche) Date: Sun, 31 Jul 2011 17:05:10 -0400 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> <87ei16dxq7.fsf@pobox.com> Message-ID: <878vreduux.fsf@pobox.com> Janzert Sun, 31 Jul 2011 16:18:32 -0400 > The issue with permissions is supposed to already be fixed in trunk Great! > https://bugs.pypy.org/issue772 > so the next release should resolve that problem for system wide installs. By "next release," do you mean 1.6? (BTW, to show bona fides, and since it needed done, I updated http://wiki.python.org/moin/PyPy to include 1.5, recycling material from the blog.) For when is that planned? Tom Roche Sun, 31 Jul 2011 15:44:55 -0400 >>>> It Would Be Nice for new adopters if the download page documented >>>> [install details], esp if intentional. Maciej Fijalkowski Sun, 31 Jul 2011 21:47:16 +0200 >>> Cool, feel like providing a patch? Tom Roche Sun, 31 Jul 2011 16:03:12 -0400 >> To the build or the doc? Maciej Fijalkowski Sun, 31 Jul 2011 22:06:36 +0200 > To the doc. >> have youse thought about a wiki, or other vehicle for user-contributed doc? > We have a [public] wiki on bitbucket (apparently underused) not to mention underexposed: I had no idea it was there. (pypy.org links to readthedocs, but that's a private wiki? So you have pypy.org, the blog, readthedocs, the bitbucket wiki, ... lot to manage.) But now I see https://bitbucket.org/pypy/pypy/wiki/Home I could put something in there, and you could point to that? From fijall at gmail.com Sun Jul 31 23:13:17 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 23:13:17 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <878vreduux.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> <87ei16dxq7.fsf@pobox.com> <878vreduux.fsf@pobox.com> Message-ID: On Sun, Jul 31, 2011 at 11:05 PM, Tom Roche wrote: > > Janzert Sun, 31 Jul 2011 16:18:32 -0400 >> The issue with permissions is supposed to already be fixed in trunk > > Great! > >> https://bugs.pypy.org/issue772 > >> so the next release should resolve that problem for system wide installs. > > By "next release," do you mean 1.6? (BTW, to show bona fides, and since it needed done, I updated > > http://wiki.python.org/moin/PyPy > > to include 1.5, recycling material from the blog.) For when is that planned? > > Tom Roche Sun, 31 Jul 2011 15:44:55 -0400 >>>>> It Would Be Nice for new adopters if the download page documented >>>>> [install details], esp if intentional. > > Maciej Fijalkowski Sun, 31 Jul 2011 21:47:16 +0200 >>>> Cool, feel like providing a patch? > > Tom Roche Sun, 31 Jul 2011 16:03:12 -0400 >>> To the build or the doc? > > Maciej Fijalkowski Sun, 31 Jul 2011 22:06:36 +0200 >> To the doc. > >>> have youse thought about a wiki, or other vehicle for user-contributed doc? > >> We have a [public] wiki on bitbucket (apparently underused) > > not to mention underexposed: I had no idea it was there. (pypy.org links to readthedocs, but that's a private wiki? So you have pypy.org, the blog, readthedocs, the bitbucket wiki, ... lot to manage.) ?But now I see > > https://bitbucket.org/pypy/pypy/wiki/Home > > I could put something in there, and you could point to that? I guess this wiki is a bit half-hearted. Ideally we would like a diff from pypy.org download page to what you want to put info there. pypy.org source is on bitbucket (in pypy/pypy.org). From fijall at gmail.com Sun Jul 31 23:14:02 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 Jul 2011 23:14:02 +0200 Subject: [pypy-dev] newbie needs pypy setup help In-Reply-To: <878vreduux.fsf@pobox.com> References: <874o23fm69.fsf@pobox.com> <871ux7ffwt.fsf@pobox.com> <20110731022749.2280.942749776.divmod.xquotient.228@localhost.localdomain> <87mxfue0hl.fsf@pobox.com> <87hb62dyko.fsf@pobox.com> <87ei16dxq7.fsf@pobox.com> <878vreduux.fsf@pobox.com> Message-ID: > By "next release," do you mean 1.6? (BTW, to show bona fides, and since it needed done, I updated > > http://wiki.python.org/moin/PyPy > > to include 1.5, recycling material from the blog.) For when is that planned? > "Soon". Feature freeze is probably tomorrow