From fijall at gmail.com Wed Nov 2 07:30:56 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 2 Nov 2011 07:30:56 +0100 Subject: [pypy-dev] Late for sprint Message-ID: hi I'm a bit late for sprint, since there was an accident on warsaw airport and most flights are cancelled. ETA tomorrow around 2pm Cheers, fijal PS. Don't wait for me with dinner :) From mark.pearse at skynet.be Wed Nov 2 10:37:34 2011 From: mark.pearse at skynet.be (mark.pearse at skynet.be) Date: Wed, 02 Nov 2011 10:37:34 +0100 Subject: [pypy-dev] Oops! Message-ID: <9bd84b$hnejk0@privrelay-pool1.isp.belgacom.be> Hi Fellow Sprinters, Only when I got to Copenhagen airport and started doing a bit of preparation did I remember that the old Powerbook that am bringing has PPC architecture. I assume this will be impractical for use at the sprint, so has someone got a spare x86 machine, preferably linux or Mac that I could borrow for the sprint? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed Nov 2 11:26:50 2011 From: arigo at tunes.org (Armin Rigo) Date: Wed, 2 Nov 2011 11:26:50 +0100 Subject: [pypy-dev] Oops! In-Reply-To: <9bd84b$hnejk0@privrelay-pool1.isp.belgacom.be> References: <9bd84b$hnejk0@privrelay-pool1.isp.belgacom.be> Message-ID: Hi Mark, On Wed, Nov 2, 2011 at 10:37, wrote: > I assume this will be impractical for use at the sprint, so has someone got > a spare x86 machine, preferably linux or Mac that I could borrow for the > sprint? That should not be a big problem. You can do a lot of development with any machine --- and then log in on some fast server if you occasionally need to translate. This is what I often do even with my modern x86 laptop, just because the server is even faster (and has more cores). Also, a sprint is a lot about peer-programming, which divides the number of needed laptops by two :-) Have a nice 2nd-half-of-travel, Armin. From lightdarkadmin at 163.com Fri Nov 4 03:41:28 2011 From: lightdarkadmin at 163.com (HY) Date: Fri, 4 Nov 2011 10:41:28 +0800 (CST) Subject: [pypy-dev] BUG! file.readlines Message-ID: <21f58a0b.10c32.1336c73ce85.Coremail.lightdarkadmin@163.com> ????, ???? windows ?, pypy ?????, ??????, ?????? CPU, ??????, ????? 20 ????????. Cpython ?????? ????????, ??. >>> f01= file('aaa.txt', 'r' ) >>> a01= f01.readlines(); #???? >>> f01.close() -------------- next part -------------- An HTML attachment was scrubbed... URL: From zzz654321 at gmail.com Fri Nov 4 05:15:57 2011 From: zzz654321 at gmail.com (ZZZ 654321) Date: Fri, 4 Nov 2011 12:15:57 +0800 Subject: [pypy-dev] BUG! file.readlines Message-ID: ????, ???? windows ?, pypy ?????, ??????, ?????? CPU, ??????, ????? 20 ????????. Cpython ?????? ????????, ??. >>> f01= file('aaa.txt', 'r' ) >>> a01= f01.readlines(); #???? >>> f01.close() From davide.setti at gmail.com Fri Nov 4 08:31:50 2011 From: davide.setti at gmail.com (Davide Setti) Date: Fri, 4 Nov 2011 08:31:50 +0100 Subject: [pypy-dev] 404 and DEBUG=True on speed.pypy.org Message-ID: Hi, If i click on the permalink link on http://speed.pypy.org/timeline/#/?exe=3,6,1,5&base=2+472&ben=grid&env=1&revs=200&equid=off i get: http://speed.pypy.org/timeline/?exe=3%2C6%2C1%2C5&base=2%2B472&ben=grid&env=1&revs=200&equid=off DEBUG=True is not good... Regards. -- Davide Setti code: http://github.com/vad From tobami at googlemail.com Fri Nov 4 09:02:11 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 4 Nov 2011 09:02:11 +0100 Subject: [pypy-dev] 404 and DEBUG=True on speed.pypy.org In-Reply-To: References: Message-ID: Thanks Davide. It's fixed now. Cheers Miquel 2011/11/4 Davide Setti : > Hi, > > If i click on the permalink link on > > http://speed.pypy.org/timeline/#/?exe=3,6,1,5&base=2+472&ben=grid&env=1&revs=200&equid=off > > i get: > > http://speed.pypy.org/timeline/?exe=3%2C6%2C1%2C5&base=2%2B472&ben=grid&env=1&revs=200&equid=off > > DEBUG=True is not good... > > Regards. > -- > > Davide Setti > code: http://github.com/vad > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From amauryfa at gmail.com Fri Nov 4 09:58:28 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 4 Nov 2011 09:58:28 +0100 Subject: [pypy-dev] BUG! file.readlines In-Reply-To: References: Message-ID: If there are people like me who can't read Chinese, an automatic translation suggests that with any version of pypy (on Windows), readlines() is very slow and consumes a lot of CPU; the sample file was 20M. 2011/11/4 ZZZ 654321 > ????, ???? windows ?, pypy ?????, ??????, ?????? CPU, ??????, ????? > 20 ????????. Cpython ?????? > ????????, ??. > > >>> f01= file('aaa.txt', 'r' ) > >>> a01= f01.readlines(); #???? > >>> f01.close() > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri Nov 4 12:18:13 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 4 Nov 2011 12:18:13 +0100 Subject: [pypy-dev] BUG! file.readlines In-Reply-To: References: Message-ID: Hi, I think I improved the time quite a lot in revision c98931f5191b. The memory usage is known to be larger on PyPy for this kind of operations. For a more general program with a lot of instances it can be equivalent or slightly better than CPython. A bient?t, Armin. From anto.cuni at gmail.com Mon Nov 7 19:04:02 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 07 Nov 2011 19:04:02 +0100 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: References: Message-ID: <4EB81D92.3040909@gmail.com> Hello Ian, On 25/07/11 11:00, Ian Ozsvald wrote: > Dear all, I've published v0.2 of my High Performance Python tutorial > write-up from the session I ran at EuroPython: > http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from-europython-2011/ today I and Armin investigated a bit more about the performances of the mandelbrot algorithm that you wrote for your tutorial. What we found is very interesting :-). We compared three versions of the code: - a (slightly modified) pure python one on PyPy - the Cython one using calculate_z.pyx_2_bettermath - the shedskin one, using shedskin2.py The PyPy version looks like this: def calculate_z_serial_purepython(q, maxiter, z): """Pure python with complex datatype, iterating over list of q and z""" output = [0] * len(q) for i in range(len(q)): zi = z[i] qi = q[i] for iteration in range(maxiter): zi = zi * zi + qi if (zi.real*zi.real + zi.imag*zi.imag) > 4.0: output[i] = iteration break return output i.e., it is exactly the same as pure_python_2.py, but we avoid to use abs(zi), so it is comparable with the cython and shedskin version. First, we ran the programs to calculate passing "1000 1000" as arguments, and these are the results: PyPy: 1.95 secs Cython: 0.58 secs Shedskin: 0.42 secs so, PyPy is ~4.5x slower than Shedskin. However, we realized that using the default values for x1,x2,y1,y2, the innermost loop runs very few iterations most of the time, and this is one case in which PyPy suffer most, because it needs to go through a bridge to continue the execution, and at the moment bridges are slower than loops. So, we changed the values of x1,x2,y1,y2 to compute a different region, in which the innermost loop runs more frequently. We used these values: x1, x2, y1, y2 = 0.37865401-0.02, 0.37865401+0.02, 0.669227668-0.02, 0.669227668+0.02 and since all programs are faster to compute the image, we used "3000 3000" as arguments from the command line. These are the results: PyPy: 0.89 Cython: 1.76 Shedskin: 0.26 So, in this case, PyPy is ~2x faster than Cython and ~3.5x slower than Shedskin. In the meantime, Armin wrote a C version of it: http://paste.pocoo.org/raw/504216/ which tooks 0.946 seconds to complete. This is in line with the PyPy's result, but we are still investigating why the shedskin's version is so much faster. ciao, Anto From harry at resolversystems.com Wed Nov 9 12:18:19 2011 From: harry at resolversystems.com (Harry Percival) Date: Wed, 09 Nov 2011 11:18:19 +0000 Subject: [pypy-dev] pypy now on pythonanywhere.com :) Message-ID: <4EBA617B.8060805@resolversystems.com> Howdy PyPy developers! Just a quick announcement - we've just made pypy (1.6) available as a console type on www.pythonanywhere.com (which is a browser-based console/ide/hosting thingummy*) cheers, Harry * anyone who went to pycon uk might remember my "excitable" talk on the subject. anyone who went to europython might remember my "shockingly poor" talk on the subject... -- Harry Percival Developer harry at resolversystems.com +44 (0) 20 3051 2751 17a Clerkenwell Road, London EC1M 5RD, UK VAT No.: GB 893 5643 79 Registered in England and Wales as company number 5467329. Registered address: 843 Finchley Road, London NW11 8NA, UK From hakan at debian.org Thu Nov 10 09:29:53 2011 From: hakan at debian.org (Hakan Ardo) Date: Thu, 10 Nov 2011 09:29:53 +0100 Subject: [pypy-dev] jit-targets Message-ID: Hi, the idea behind the jit-targets branch is to generalize the concept of unrolling into something that makes sens for bridges aswell. Below I try to describe the goals of this intention. The jit-targets branch will be a first step towards this goal. First, we need a new way to model traces. A trace now consists of several parts separated by a new ResOperation called LABEL. Labels are inserted into the trace at strategic places (different startegies could be inverstigated). The label has a TargetToken descr that describes the place at which it is inserted. It includes * A list of all live variables (the label arguments) * A VirtualState describing the live virtuals * A short preamble that can be inlined to jump to the label A JUMP ResOperation now has a TargetToken as a descr and it will always jump to the label with the same TargetToken. Such a jump is only legal if the VirtualState at the jump matches the VirtualState of the TargetToken and the short preamble has been inlined. Each JitCellToken will contain a list of TargeTokens that corresponds to the same greenkey (but with different VirtualStates). When we produce a new trace, we will trace until we discover a JitCell with a JitCellToken. The VirtualState at the end of the trace is comapred to the VirtualState of all the TargetTokens listed. If one of them matches, it's short preamble is inlined to produce a jump to it. If none matches we insert a label into the trace and continue tracing. To handle traces that loop back on themselfs we make sure that a label is inserted at the merge point. For loops this will produce the same effect as unrolling, but with twice as much tracing as both the first and second iterations are traced. We can however detect such loops and inline the produced trace instead of continue traceing without chaning this new concept. Also it will make it straight forward to unroll into more than 2 iterations if that would make sens. The optimizeopt optimizer will be passed trace parts that can start and/or end with a label, but there with be no intermediate labels. If the trace-part ends with a label, the state of the optimizer that would previously be passed from the preamble to the loop will now instead be saved on this label. If the trace-part starts with a label the corresponding state will be imported from it. If a box produced in one trace-part is reused in the following trace-part it will be added as an argument to the intermediate label. Also resops to produce the box will be added to the short preamble of that label. The logic here on how to import boxes from previous trace-parts is the same as what was previous used to import boxes from the preamble into the loop. The jit-targets branch implementes the framework described above. To not make it too big a leap from where we are today it then uses a policy of where to insert labels and when to force virtuals that will make it produce as similiar results as possible to what we have today. That becomes a somewhat complex policy. I do however believe that simpler policies (such as what's hinted about above) will be even more effective here. That would allow for things to be simplified and cleaned up a bit. Also, experimenting with inserting labels at say the first jit_merge_point discovered when tracing a bridge would allow us to join the controllflows of a loop with lots of paths back together. That will however probably be the topic of future branches... -- H?kan Ard? From fijall at gmail.com Thu Nov 10 10:38:44 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 10 Nov 2011 10:38:44 +0100 Subject: [pypy-dev] pypy now on pythonanywhere.com :) In-Reply-To: <4EBA617B.8060805@resolversystems.com> References: <4EBA617B.8060805@resolversystems.com> Message-ID: On Wed, Nov 9, 2011 at 12:18 PM, Harry Percival wrote: > Howdy PyPy developers! ?Just a quick announcement - we've just made pypy > (1.6) available as a console type on www.pythonanywhere.com (which is a > browser-based console/ide/hosting thingummy*) > > > > cheers, > Harry Hi Cool! Is there a possibility we can play with it a bit to see how it works? Cheers, fijal From arkanosis at gmail.com Thu Nov 10 10:52:56 2011 From: arkanosis at gmail.com (=?UTF-8?B?SsOpcsOpbWllIFJvcXVldA==?=) Date: Thu, 10 Nov 2011 10:52:56 +0100 Subject: [pypy-dev] pypy now on pythonanywhere.com :) In-Reply-To: References: <4EBA617B.8060805@resolversystems.com> Message-ID: Hi Maciej, 2011/11/10 Maciej Fijalkowski : > Cool! Is there a possibility we can play with it a bit to see how it works? I've just signed up and I can test everything. BTW, truly awesome :-) Best regards, -- J?r?mie From harry at resolversystems.com Thu Nov 10 11:58:59 2011 From: harry at resolversystems.com (Harry Percival) Date: Thu, 10 Nov 2011 10:58:59 +0000 Subject: [pypy-dev] pypy now on pythonanywhere.com :) In-Reply-To: References: <4EBA617B.8060805@resolversystems.com> Message-ID: <4EBBAE73.4010206@resolversystems.com> Hey Maciej, Like J?r?mie, you can just sign up, and you should get an invitation pretty quick - just drop me an email if there's any kind of hangup and I'll get you bumped up to the top of the list. Same goes for anyone on the list really. One thing we did for the IPython guys was to give them a "try IPython" link, which lets anonymous users try out an IPython console without needing to sign up - see http://www.pythonanywhere.com/try-ipython/ Maybe we could do something similar for PyPy? try-pypy? Thinking out loud, maybe it's less interesting for users to try out an pypy console - compared to the IPython one, which has loads of presentational features, the pypy one is just like a normal python console... On the other hand, maybe it would be a useful thing for people who want to reassure themselves that it really does work just like normal python?? Up to you guys! cheers, Harry On 10/11/11 09:52, J?r?mie Roquet wrote: > Hi Maciej, > > 2011/11/10 Maciej Fijalkowski : >> Cool! Is there a possibility we can play with it a bit to see how it works? > > I've just signed up and I can test everything. > > BTW, truly awesome :-) > > Best regards, > -- Harry Percival Developer harry at resolversystems.com +44 (0) 20 3051 2751 Dirigible: a Python cloud spreadsheet 17a Clerkenwell Road, London EC1M 5RD, UK VAT No.: GB 893 5643 79 Registered in England and Wales as company number 5467329. Registered address: 843 Finchley Road, London NW11 8NA, UK From cfbolz at gmx.de Tue Nov 15 15:11:35 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 15 Nov 2011 15:11:35 +0100 Subject: [pypy-dev] Developer selection for Py3k and Numpy In-Reply-To: <4E9D65C0.5090607@gmx.de> References: <4E9D65C0.5090607@gmx.de> Message-ID: <4EC27317.6090206@gmx.de> On 10/18/2011 01:40 PM, Carl Friedrich Bolz wrote: > Now that we are getting in some money for our Py3k [1] and Numpy [2] > funding proposals (thank you very very much, for everybody who > contributed!) it is time to think more concretely about the actual > execution. > > Therefore I want to ask for PyPy developers that are interested in > getting paid for their work on the first steps of the Numpy or Py3k > proposals to step forward. To be applicable you need to be an > experienced PyPy developer who worked in this area before (Numpy) or on > the Python interpreter (Py3k). Hi all, wanted to follow up on this. The selected developers for the first phases are: Py3k phase 1.1: Antonio Cuni has been a core developer of PyPy for 5 years, and has worked in many areas of the project, including the Python interpreter, the backends and the JIT compiler generator. He has a deep knowledge of the source code, and thus he is well suited to do the work to implement Python 3 in PyPy. Maciej Fija?kowski is a long time PyPy core developer and has participated in the development of the Python interpreter since 2006. He helped with the move from interpreter version from Python 2.4 to 2.5 as well as 2.5 to 2.7. He also worked a lot on extension modules, like ctypes. Thus he is well suited to implement Python 3 in PyPy. Alex Gaynor has been contributing to PyPy for 1.5 years. He has worked on many parts of PyPy, including the JIT, interpreter, translator, and some modules. He helped doing the port to 2.7, including writing the _io module implementation, making him well-suited to work on the Python 3-effort. Benjamin Peterson played a major role in porting PyPy to Python 2.7. His understanding of the PyPy codebase is very broad. He is also a CPython core developer and played a crucial role in implementing Python 3 there, which makes him an ideal candidate for helping the equivalent effort in PyPy. Numpy phase 1: Maciej Fija?kowski started developing the Numpy experiments done in PyPy in 2009. He also created the first version of arrays that were used for this experiment. He is also one of the maintainers of the x86 JIT-backend which makes him suitable for work on the performance front. Cheers, Carl Friedrich From ian at ianozsvald.com Tue Nov 15 15:54:07 2011 From: ian at ianozsvald.com (Ian Ozsvald) Date: Tue, 15 Nov 2011 14:54:07 +0000 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: <4EB81D92.3040909@gmail.com> References: <4EB81D92.3040909@gmail.com> Message-ID: Hi Antonio! Apologies for the slow reply, this got filed into a subfolder. The numbers are interesting, I'm also interested in the C version. I'm hoping that my tutorial will be accepted for PyCon next March (the talks are announced in two weeks), assuming I get to talk again I'll update my tutorial. Adding more for PyPy and having a C equivalent will be very useful. Given that the C version should be very similar to the ShedSkin version, maybe it just comes down to compiler differences? On my Macbook (where I originally wrote the talk) I think the differences in speed came from two versions of gcc (Cython seemed to prefer one, ShedSkin the other, I ran out of time trying to unify that test). Do you definitely use the same optimisation flags? ShedSkin (from memory) requests fast math and a few other things in the generated Makefile. Ian. On 7 November 2011 18:04, Antonio Cuni wrote: > Hello Ian, > > On 25/07/11 11:00, Ian Ozsvald wrote: >> >> Dear all, I've published v0.2 of my High Performance Python tutorial >> write-up from the session I ran at EuroPython: >> >> http://ianozsvald.com/2011/07/25/high-performance-python-tutorial-v0-2-from-europython-2011/ > > today I and Armin investigated a bit more about the performances of the > mandelbrot algorithm that you wrote for your tutorial. ?What we found is > very interesting :-). > > We compared three versions of the code: > > - a (slightly modified) pure python one on PyPy > - the Cython one using calculate_z.pyx_2_bettermath > - the shedskin one, using shedskin2.py > > The PyPy version looks like this: > > def calculate_z_serial_purepython(q, maxiter, z): > ? ?"""Pure python with complex datatype, iterating over list of q and z""" > ? ?output = [0] * len(q) > ? ?for i in range(len(q)): > ? ? ? ?zi = z[i] > ? ? ? ?qi = q[i] > ? ? ? ?for iteration in range(maxiter): > ? ? ? ? ? ?zi = zi * zi + qi > ? ? ? ? ? ?if (zi.real*zi.real + zi.imag*zi.imag) > 4.0: > ? ? ? ? ? ? ? ?output[i] = iteration > ? ? ? ? ? ? ? ?break > ? ?return output > > i.e., it is exactly the same as pure_python_2.py, but we avoid to use > abs(zi), so it is comparable with the cython and shedskin version. > > First, we ran the programs to calculate passing "1000 1000" as arguments, > and these are the results: > > PyPy: 1.95 secs > Cython: 0.58 secs > Shedskin: 0.42 secs > > so, PyPy is ~4.5x slower than Shedskin. > > However, we realized that using the default values for x1,x2,y1,y2, the > innermost loop runs very few iterations most of the time, and this is one > case in which PyPy suffer most, because it needs to go through a bridge to > continue the execution, and at the moment bridges are slower than loops. > > So, we changed the values of x1,x2,y1,y2 to compute a different region, in > which the innermost loop runs more frequently. ?We used these values: > x1, x2, y1, y2 = 0.37865401-0.02, 0.37865401+0.02, 0.669227668-0.02, > 0.669227668+0.02 > > and since all programs are faster to compute the image, we used "3000 3000" > as arguments from the command line. ?These are the results: > > PyPy: 0.89 > Cython: 1.76 > Shedskin: 0.26 > > So, in this case, PyPy is ~2x faster than Cython and ~3.5x slower than > Shedskin. > > In the meantime, Armin wrote a C version of it: > http://paste.pocoo.org/raw/504216/ > > which tooks 0.946 seconds to complete. This is in line with the PyPy's > result, but we are still investigating why the shedskin's version is so much > faster. > > ciao, > Anto > -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://StrongSteam.com/ http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald From arigo at tunes.org Tue Nov 15 16:43:52 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 15 Nov 2011 16:43:52 +0100 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: References: <4EB81D92.3040909@gmail.com> Message-ID: Hi, On Tue, Nov 15, 2011 at 15:54, Ian Ozsvald wrote: > ShedSkin (from memory) > requests fast math and a few other things in the generated Makefile. Ah, it is cheating that way. Indeed, I didn't try to play with gcc options; I just used -O2 (or -O3, which made no difference). The C source code is completely obvious and surprise-less. You can see it here (it outputs the raw data to stdout, so you have to pipe it to a converter program to display the result): http://paste.pocoo.org/show/508215/ A bient?t, Armin. From arkanosis at gmail.com Tue Nov 15 17:11:53 2011 From: arkanosis at gmail.com (=?UTF-8?B?SsOpcsOpbWllIFJvcXVldA==?=) Date: Tue, 15 Nov 2011 17:11:53 +0100 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: References: <4EB81D92.3040909@gmail.com> Message-ID: Hi, 2011/11/15 Armin Rigo : > On Tue, Nov 15, 2011 at 15:54, Ian Ozsvald wrote: >> ShedSkin (from memory) >> requests fast math and a few other things in the generated Makefile. > > Ah, it is cheating that way. ?Indeed, I didn't try to play with gcc > options; I just used -O2 (or -O3, which made no difference). FYI, here is the default FLAGS file for shedskin: CC=g++ CCFLAGS=-O2 -march=native -Wno-deprecated $(CPPFLAGS) LFLAGS=-lgc -lpcre $(LDFLAGS) But of course you can change the compiler and play with its flags to improve performance. Best regards, -- J?r?mie From frikker at gmail.com Tue Nov 15 18:56:47 2011 From: frikker at gmail.com (Blaine) Date: Tue, 15 Nov 2011 12:56:47 -0500 Subject: [pypy-dev] Detecting pypy vs cpython runtime Message-ID: Hi everyone. I have a cellular automata framework in C++ and I use cython and cpython to run it. I found out that if I port it to pure python and run it with pypy, it's close to the same performance as the C++ version. (about 2x as slow, compared to 20x as slow when using pure python + cpython). When I throw in other overheads with pure python libraries, using the pure python and pypy is much faster than cpython with the C++ library, all things equal. What I'd like to do is detect if pypy or cpython is doing the importing of my module, and switch over to the pure python interface if pypy is found. As it stands I have to do it manually in my module's __init__.py. *Is there any way to detect if my module is being imported by pypy vs cpython?* Either via sys, or maybe some latent variable that is present, or something else. sys.argv[0] only has the script name (obviously), not the interpreter call. Keep up the outstanding work. Pypy is great! Thanks! Blaine -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Tue Nov 15 19:03:24 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 15 Nov 2011 13:03:24 -0500 Subject: [pypy-dev] Detecting pypy vs cpython runtime In-Reply-To: References: Message-ID: On Tue, Nov 15, 2011 at 12:56 PM, Blaine wrote: > Hi everyone. I have a cellular automata framework in C++ and I use cython > and cpython to run it. I found out that if I port it to pure python and run > it with pypy, it's close to the same performance as the C++ version. (about > 2x as slow, compared to 20x as slow when using pure python + cpython). When > I throw in other overheads with pure python libraries, using the pure > python and pypy is much faster than cpython with the C++ library, all > things equal. > > What I'd like to do is detect if pypy or cpython is doing the importing of > my module, and switch over to the pure python interface if pypy is found. > As it stands I have to do it manually in my module's __init__.py. > > *Is there any way to detect if my module is being imported by pypy vs > cpython?* Either via sys, or maybe some latent variable that is present, > or something else. sys.argv[0] only has the script name (obviously), not > the interpreter call. > > Keep up the outstanding work. Pypy is great! > > Thanks! > Blaine > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > The canonical way is probably `import platform; platform.python_implementation()` which will return either "PyPy" or "CPython". Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 15 19:08:25 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 15 Nov 2011 20:08:25 +0200 Subject: [pypy-dev] Detecting pypy vs cpython runtime In-Reply-To: References: Message-ID: On Tue, Nov 15, 2011 at 8:03 PM, Alex Gaynor wrote: > > > On Tue, Nov 15, 2011 at 12:56 PM, Blaine wrote: >> >> Hi everyone. I have a cellular automata framework in C++ and I use cython >> and cpython to run it. I found out that if I port it to pure python and run >> it with pypy, it's close to the same performance as the C++ version. (about >> 2x as slow, compared to 20x as slow when using pure python + cpython). When >> I throw in other overheads with pure python libraries, using the pure python >> and pypy is much faster than cpython with the C++ library, all things equal. >> What I'd like to do is detect if pypy or cpython is doing the importing of >> my module, and switch over to the pure python interface if pypy is found. As >> it stands I have to do it manually in my module's __init__.py. >> Is there any way to detect if my module is being imported by pypy vs >> cpython??Either via sys, or maybe some latent variable that is present, or >> something else. sys.argv[0] only has the script name (obviously), not the >> interpreter call. >> Keep up the outstanding work. Pypy is great! >> Thanks! >> Blaine >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > > The canonical way is probably `import platform; > platform.python_implementation()` which will return either "PyPy" or > "CPython". > Alex If you need to support older pythons (which don't have platform.python_implementation), we use import sys is_pypy = '__pypy__' in sys.builtin_module_names or alternatively: try: import __pypy__ except ImportError: __pypy__ = None > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > From alex.gaynor at gmail.com Tue Nov 15 19:10:30 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 15 Nov 2011 13:10:30 -0500 Subject: [pypy-dev] Detecting pypy vs cpython runtime In-Reply-To: References: Message-ID: On Tue, Nov 15, 2011 at 1:08 PM, Maciej Fijalkowski wrote: > On Tue, Nov 15, 2011 at 8:03 PM, Alex Gaynor > wrote: > > > > > > On Tue, Nov 15, 2011 at 12:56 PM, Blaine wrote: > >> > >> Hi everyone. I have a cellular automata framework in C++ and I use > cython > >> and cpython to run it. I found out that if I port it to pure python and > run > >> it with pypy, it's close to the same performance as the C++ version. > (about > >> 2x as slow, compared to 20x as slow when using pure python + cpython). > When > >> I throw in other overheads with pure python libraries, using the pure > python > >> and pypy is much faster than cpython with the C++ library, all things > equal. > >> What I'd like to do is detect if pypy or cpython is doing the importing > of > >> my module, and switch over to the pure python interface if pypy is > found. As > >> it stands I have to do it manually in my module's __init__.py. > >> Is there any way to detect if my module is being imported by pypy vs > >> cpython? Either via sys, or maybe some latent variable that is present, > or > >> something else. sys.argv[0] only has the script name (obviously), not > the > >> interpreter call. > >> Keep up the outstanding work. Pypy is great! > >> Thanks! > >> Blaine > >> > >> _______________________________________________ > >> pypy-dev mailing list > >> pypy-dev at python.org > >> http://mail.python.org/mailman/listinfo/pypy-dev > >> > > > > The canonical way is probably `import platform; > > platform.python_implementation()` which will return either "PyPy" or > > "CPython". > > Alex > > If you need to support older pythons (which don't have > platform.python_implementation), we use > > import sys > is_pypy = '__pypy__' in sys.builtin_module_names > > or alternatively: > > try: > import __pypy__ > except ImportError: > __pypy__ = None > > > > > -- > > "I disapprove of what you say, but I will defend to the death your right > to > > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > > "The people's good is the highest law." -- Cicero > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > > > > > And don't forget `hasattr(sys, "pypy_translation_info")` just for completeness, platform is the cleanest IMO though :) Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From frikker at gmail.com Tue Nov 15 19:16:25 2011 From: frikker at gmail.com (Blaine) Date: Tue, 15 Nov 2011 13:16:25 -0500 Subject: [pypy-dev] Detecting pypy vs cpython runtime In-Reply-To: References: Message-ID: Awesome. Thanks everyone, exactly what I was looking for. Thank you! Blaine On Tue, Nov 15, 2011 at 1:10 PM, Alex Gaynor wrote: > > > On Tue, Nov 15, 2011 at 1:08 PM, Maciej Fijalkowski wrote: > >> On Tue, Nov 15, 2011 at 8:03 PM, Alex Gaynor >> wrote: >> > >> > >> > On Tue, Nov 15, 2011 at 12:56 PM, Blaine wrote: >> >> >> >> Hi everyone. I have a cellular automata framework in C++ and I use >> cython >> >> and cpython to run it. I found out that if I port it to pure python >> and run >> >> it with pypy, it's close to the same performance as the C++ version. >> (about >> >> 2x as slow, compared to 20x as slow when using pure python + cpython). >> When >> >> I throw in other overheads with pure python libraries, using the pure >> python >> >> and pypy is much faster than cpython with the C++ library, all things >> equal. >> >> What I'd like to do is detect if pypy or cpython is doing the >> importing of >> >> my module, and switch over to the pure python interface if pypy is >> found. As >> >> it stands I have to do it manually in my module's __init__.py. >> >> Is there any way to detect if my module is being imported by pypy vs >> >> cpython? Either via sys, or maybe some latent variable that is >> present, or >> >> something else. sys.argv[0] only has the script name (obviously), not >> the >> >> interpreter call. >> >> Keep up the outstanding work. Pypy is great! >> >> Thanks! >> >> Blaine >> >> >> >> _______________________________________________ >> >> pypy-dev mailing list >> >> pypy-dev at python.org >> >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> >> > >> > The canonical way is probably `import platform; >> > platform.python_implementation()` which will return either "PyPy" or >> > "CPython". >> > Alex >> >> If you need to support older pythons (which don't have >> platform.python_implementation), we use >> >> import sys >> is_pypy = '__pypy__' in sys.builtin_module_names >> >> or alternatively: >> >> try: >> import __pypy__ >> except ImportError: >> __pypy__ = None >> >> > >> > -- >> > "I disapprove of what you say, but I will defend to the death your >> right to >> > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) >> > "The people's good is the highest law." -- Cicero >> > >> > >> > _______________________________________________ >> > pypy-dev mailing list >> > pypy-dev at python.org >> > http://mail.python.org/mailman/listinfo/pypy-dev >> > >> > >> > > And don't forget `hasattr(sys, "pypy_translation_info")` just for > completeness, platform is the cleanest IMO though :) > > Alex > > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Nov 16 08:05:32 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 16 Nov 2011 09:05:32 +0200 Subject: [pypy-dev] [pypy-commit] pypy ppc-jit-backend: setarrayitem and getarrayitem offsets are immediate values. In-Reply-To: <20111114192726.8DC30820BE@wyvern.cs.uni-duesseldorf.de> References: <20111114192726.8DC30820BE@wyvern.cs.uni-duesseldorf.de> Message-ID: Unless I'm missing something offset for get/setarrayitem are not necesarilly intermediate values On Mon, Nov 14, 2011 at 9:27 PM, edelsohn wrote: > Author: edelsohn > Branch: ppc-jit-backend > Changeset: r49414:3a6600bf032a > Date: 2011-11-14 14:27 -0500 > http://bitbucket.org/pypy/pypy/changeset/3a6600bf032a/ > > Log: ? ?setarrayitem and getarrayitem offsets are immediate values. > > diff --git a/pypy/jit/backend/ppc/ppcgen/opassembler.py b/pypy/jit/backend/ppc/ppcgen/opassembler.py > --- a/pypy/jit/backend/ppc/ppcgen/opassembler.py > +++ b/pypy/jit/backend/ppc/ppcgen/opassembler.py > @@ -367,11 +367,10 @@ > ? ? ? ? value_loc, base_loc, ofs_loc, scale, ofs = arglocs > ? ? ? ? if scale.value > 0: > ? ? ? ? ? ? scale_loc = r.r0 > - ? ? ? ? ? ?self.mc.load_imm(r.r0, scale.value) > ? ? ? ? ? ? if IS_PPC_32: > - ? ? ? ? ? ? ? ?self.mc.slw(r.r0.value, ofs_loc.value, r.r0.value) > + ? ? ? ? ? ? ? ?self.mc.slwi(r.r0.value, ofs_loc.value, scale.value) > ? ? ? ? ? ? else: > - ? ? ? ? ? ? ? ?self.mc.sld(r.r0.value, ofs_loc.value, r.r0.value) > + ? ? ? ? ? ? ? ?self.mc.sldi(r.r0.value, ofs_loc.value, scale.value) > ? ? ? ? else: > ? ? ? ? ? ? scale_loc = ofs_loc > > @@ -396,11 +395,10 @@ > ? ? ? ? res, base_loc, ofs_loc, scale, ofs = arglocs > ? ? ? ? if scale.value > 0: > ? ? ? ? ? ? scale_loc = r.r0 > - ? ? ? ? ? ?self.mc.load_imm(r.r0, scale.value) > ? ? ? ? ? ? if IS_PPC_32: > - ? ? ? ? ? ? ? ?self.mc.slw(r.r0.value, ofs_loc.value, scale.value) > + ? ? ? ? ? ? ? ?self.mc.slwi(r.r0.value, ofs_loc.value, scale.value) > ? ? ? ? ? ? else: > - ? ? ? ? ? ? ? ?self.mc.sld(r.r0.value, ofs_loc.value, scale.value) > + ? ? ? ? ? ? ? ?self.mc.sldi(r.r0.value, ofs_loc.value, scale.value) > ? ? ? ? else: > ? ? ? ? ? ? scale_loc = ofs_loc > ? ? ? ? if ofs.value > 0: > _______________________________________________ > pypy-commit mailing list > pypy-commit at python.org > http://mail.python.org/mailman/listinfo/pypy-commit > From arigo at tunes.org Wed Nov 16 09:47:33 2011 From: arigo at tunes.org (Armin Rigo) Date: Wed, 16 Nov 2011 09:47:33 +0100 Subject: [pypy-dev] [pypy-commit] pypy ppc-jit-backend: setarrayitem and getarrayitem offsets are immediate values. In-Reply-To: References: <20111114192726.8DC30820BE@wyvern.cs.uni-duesseldorf.de> Message-ID: Hi Maciej, On Wed, Nov 16, 2011 at 08:05, Maciej Fijalkowski wrote: > Unless I'm missing something offset for get/setarrayitem are not > necesarilly intermediate values Assuming you mean "immediate" values instead of "intermediate": the content of the checkin shows that it meant that the *scale* is a constant (address = base + baseoffset + item * scale). A bient?t, Armin. From ian at ianozsvald.com Wed Nov 16 15:25:08 2011 From: ian at ianozsvald.com (Ian Ozsvald) Date: Wed, 16 Nov 2011 14:25:08 +0000 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: References: <4EB81D92.3040909@gmail.com> Message-ID: >From memory the 'native' flag made a difference (I think it allows use of SSE?). I guess that is something I'll normalise for a future v0.3 release of my handbook :-) Cheers, Ian. 2011/11/15 J?r?mie Roquet : > Hi, > > 2011/11/15 Armin Rigo : >> On Tue, Nov 15, 2011 at 15:54, Ian Ozsvald wrote: >>> ShedSkin (from memory) >>> requests fast math and a few other things in the generated Makefile. >> >> Ah, it is cheating that way. ?Indeed, I didn't try to play with gcc >> options; I just used -O2 (or -O3, which made no difference). > > FYI, here is the default FLAGS file for shedskin: > > CC=g++ > CCFLAGS=-O2 -march=native -Wno-deprecated $(CPPFLAGS) > LFLAGS=-lgc -lpcre $(LDFLAGS) > > But of course you can change the compiler and play with its flags to > improve performance. > > Best regards, > > -- > J?r?mie > -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://StrongSteam.com/ http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald From arkanosis at gmail.com Wed Nov 16 16:40:24 2011 From: arkanosis at gmail.com (=?UTF-8?B?SsOpcsOpbWllIFJvcXVldA==?=) Date: Wed, 16 Nov 2011 16:40:24 +0100 Subject: [pypy-dev] Updated 'High Performance Python' tutorial (the one from EuroPython 2011) In-Reply-To: References: <4EB81D92.3040909@gmail.com> Message-ID: 2011/11/16 Ian Ozsvald : > From memory the 'native' flag made a difference (I think it allows use > of SSE?). It depends on the machine, of course, but yes, on most machines it enables SSE. Just compare the output of $ < /dev/null g++ -E -v - |& grep cc1 and $ < /dev/null g++ -march=native -E -v - |& grep cc1 The following flags are added for me : -march=core2 -mcx16 -msahf --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=4096 -mtune=core2 -mcx16 and -msahf enable some additional instructions (see http://gcc.gnu.org/onlinedocs/gcc/i386-and-x86_002d64-Options.html) -mtune=core2 enables Intel's 64-bit extensions, MMX, SSE, SSE2, SSE3 and SSSE3 Best regards, -- J?r?mie From frikker at gmail.com Wed Nov 16 18:21:27 2011 From: frikker at gmail.com (Blaine) Date: Wed, 16 Nov 2011 12:21:27 -0500 Subject: [pypy-dev] Pure Python Math Library Message-ID: Does anyone know of a pure python math library? I've been playing around with berp , which is a python3 to haskell translator and compiler, and it works great as long as you don't go crazy with C extensions. It's highly experimental but fun to play around with. The only thing that I really miss is being able to use the math module. I asked the maintainer if it is possible to map into haskell's math library, but in the mean time a pure python math library would fit nicely since it would be compiled along with the rest of the python. I'm looking for log, log10, ceil, and pow mostly for my personal needs. It's funny now that things like pypy and berp exist. I find myself trying to locate pure python routines (like DFT) that would have no reason to exist with cpython, but make lots of sense with pypy. Anyway, just wondering if anyone had heard of such a thing. How do you even implement log? Thanks, Blaine -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Nov 16 18:25:12 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 16 Nov 2011 12:25:12 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: 2011/11/16 Blaine : > Does anyone know of a pure python math library? I've been playing around > with berp, which is a python3 to haskell translator and compiler, and it > works great as long as you don't go crazy with C extensions. It's highly > experimental but fun to play around with. The only thing that I really miss > is being able to use the math module. I asked the maintainer if it is > possible to map into haskell's math library, but in the mean time a pure > python math library would fit nicely since it would be compiled along with > the rest of the python. > I'm looking for log, log10, ceil, and pow mostly for my personal needs. Well, python has a math module too: >>> import math >>> math.log(23) 3.1354942159291497 -- Regards, Benjamin From frikker at gmail.com Wed Nov 16 18:26:13 2011 From: frikker at gmail.com (Blaine) Date: Wed, 16 Nov 2011 12:26:13 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: I think that python's math module (which I use) is a compiled C extension, right? I'm looking for pure python that berp can use. Blaine On Wed, Nov 16, 2011 at 12:25 PM, Benjamin Peterson wrote: > 2011/11/16 Blaine : > > Does anyone know of a pure python math library? I've been playing around > > with berp, which is a python3 to haskell translator and compiler, and it > > works great as long as you don't go crazy with C extensions. It's highly > > experimental but fun to play around with. The only thing that I really > miss > > is being able to use the math module. I asked the maintainer if it is > > possible to map into haskell's math library, but in the mean time a pure > > python math library would fit nicely since it would be compiled along > with > > the rest of the python. > > I'm looking for log, log10, ceil, and pow mostly for my personal needs. > > Well, python has a math module too: > > >>> import math > >>> math.log(23) > 3.1354942159291497 > > > > -- > Regards, > Benjamin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Nov 16 18:27:24 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 16 Nov 2011 12:27:24 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: 2011/11/16 Blaine : > I think that python's math module (which I use) is a compiled C extension, > right? I'm looking for pure python that berp can use. I'm not really sure why this is relevant to pypy. -- Regards, Benjamin From alex.gaynor at gmail.com Wed Nov 16 18:29:50 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 16 Nov 2011 12:29:50 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: On Wed, Nov 16, 2011 at 12:26 PM, Blaine wrote: > I think that python's math module (which I use) is a compiled C extension, > right? I'm looking for pure python that berp can use. > > Blaine > > > > On Wed, Nov 16, 2011 at 12:25 PM, Benjamin Peterson wrote: > >> 2011/11/16 Blaine : >> > Does anyone know of a pure python math library? I've been playing around >> > with berp, which is a python3 to haskell translator and compiler, and it >> > works great as long as you don't go crazy with C extensions. It's highly >> > experimental but fun to play around with. The only thing that I really >> miss >> > is being able to use the math module. I asked the maintainer if it is >> > possible to map into haskell's math library, but in the mean time a pure >> > python math library would fit nicely since it would be compiled along >> with >> > the rest of the python. >> > I'm looking for log, log10, ceil, and pow mostly for my personal needs. >> >> Well, python has a math module too: >> >> >>> import math >> >>> math.log(23) >> 3.1354942159291497 >> >> >> >> -- >> Regards, >> Benjamin >> > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > It's a part of the standard library, it's implemented however the Python VM provides it. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.j.a.cock at googlemail.com Wed Nov 16 18:33:15 2011 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Nov 2011 17:33:15 +0000 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: On Wed, Nov 16, 2011 at 5:27 PM, Benjamin Peterson wrote: > 2011/11/16 Blaine : >> I think that python's math module (which I use) is a compiled C extension, >> right? I'm looking for pure python that berp can use. > > I'm not really sure why this is relevant to pypy. > How does pypy implement the math module? It that pure Python? Peter From frikker at gmail.com Wed Nov 16 18:32:50 2011 From: frikker at gmail.com (Blaine) Date: Wed, 16 Nov 2011 12:32:50 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: Thanks Alex and Benjamin. I'm sorry - you're right it isn't exactly related to pypy. I hope I didn't break any rules. I was hoping that someone else may have come across this because the only time I've needed to port compiled modules to python is when I wanted to use them with pypy. Blaine On Wed, Nov 16, 2011 at 12:29 PM, Alex Gaynor wrote: > > > On Wed, Nov 16, 2011 at 12:26 PM, Blaine wrote: > >> I think that python's math module (which I use) is a compiled C >> extension, right? I'm looking for pure python that berp can use. >> >> Blaine >> >> >> >> On Wed, Nov 16, 2011 at 12:25 PM, Benjamin Peterson wrote: >> >>> 2011/11/16 Blaine : >>> > Does anyone know of a pure python math library? I've been playing >>> around >>> > with berp, which is a python3 to haskell translator and compiler, and >>> it >>> > works great as long as you don't go crazy with C extensions. It's >>> highly >>> > experimental but fun to play around with. The only thing that I really >>> miss >>> > is being able to use the math module. I asked the maintainer if it is >>> > possible to map into haskell's math library, but in the mean time a >>> pure >>> > python math library would fit nicely since it would be compiled along >>> with >>> > the rest of the python. >>> > I'm looking for log, log10, ceil, and pow mostly for my personal needs. >>> >>> Well, python has a math module too: >>> >>> >>> import math >>> >>> math.log(23) >>> 3.1354942159291497 >>> >>> >>> >>> -- >>> Regards, >>> Benjamin >>> >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> > It's a part of the standard library, it's implemented however the Python > VM provides it. > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Nov 16 18:36:05 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 16 Nov 2011 12:36:05 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: 2011/11/16 Blaine : > Thanks Alex and Benjamin. > ? I'm sorry - you're right it isn't exactly related to pypy. I hope I didn't > break any rules. I was hoping that someone else may have come across this > because the only time I've needed to port compiled modules to python is when > I wanted to use them with pypy. > Blaine I don't PyPy will be very helpful to you, since it uses the libc math functions. -- Regards, Benjamin From frikker at gmail.com Wed Nov 16 18:38:41 2011 From: frikker at gmail.com (Blaine) Date: Wed, 16 Nov 2011 12:38:41 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: I imagine pypy uses libc math through ctypes, since pypy and ctypes play well together. Thanks for all the tips, I'll probably just wait to see what the maintainer thinks about mapping to haskell's math library. Blaine On Wed, Nov 16, 2011 at 12:36 PM, Benjamin Peterson wrote: > 2011/11/16 Blaine : > > Thanks Alex and Benjamin. > > I'm sorry - you're right it isn't exactly related to pypy. I hope I > didn't > > break any rules. I was hoping that someone else may have come across this > > because the only time I've needed to port compiled modules to python is > when > > I wanted to use them with pypy. > > Blaine > > I don't PyPy will be very helpful to you, since it uses the libc math > functions. > > > > -- > Regards, > Benjamin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Wed Nov 16 18:40:47 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 16 Nov 2011 12:40:47 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: On Wed, Nov 16, 2011 at 12:38 PM, Blaine wrote: > I imagine pypy uses libc math through ctypes, since pypy and ctypes play > well together. > > Thanks for all the tips, I'll probably just wait to see what the > maintainer thinks about mapping to haskell's math library. > Blaine > > > > On Wed, Nov 16, 2011 at 12:36 PM, Benjamin Peterson wrote: > >> 2011/11/16 Blaine : >> > Thanks Alex and Benjamin. >> > I'm sorry - you're right it isn't exactly related to pypy. I hope I >> didn't >> > break any rules. I was hoping that someone else may have come across >> this >> > because the only time I've needed to port compiled modules to python is >> when >> > I wanted to use them with pypy. >> > Blaine >> >> I don't PyPy will be very helpful to you, since it uses the libc math >> functions. >> >> >> >> -- >> Regards, >> Benjamin >> > > No, PyPy has an RPython math module which calls libc. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From frikker at gmail.com Wed Nov 16 18:52:17 2011 From: frikker at gmail.com (Blaine) Date: Wed, 16 Nov 2011 12:52:17 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: Oh, neat. I didn't know that. Thank you for the information, I didn't know that was possible. Blaine On Wed, Nov 16, 2011 at 12:40 PM, Alex Gaynor wrote: > > > On Wed, Nov 16, 2011 at 12:38 PM, Blaine wrote: > >> I imagine pypy uses libc math through ctypes, since pypy and ctypes play >> well together. >> >> Thanks for all the tips, I'll probably just wait to see what the >> maintainer thinks about mapping to haskell's math library. >> Blaine >> >> >> >> On Wed, Nov 16, 2011 at 12:36 PM, Benjamin Peterson wrote: >> >>> 2011/11/16 Blaine : >>> > Thanks Alex and Benjamin. >>> > I'm sorry - you're right it isn't exactly related to pypy. I hope I >>> didn't >>> > break any rules. I was hoping that someone else may have come across >>> this >>> > because the only time I've needed to port compiled modules to python >>> is when >>> > I wanted to use them with pypy. >>> > Blaine >>> >>> I don't PyPy will be very helpful to you, since it uses the libc math >>> functions. >>> >>> >>> >>> -- >>> Regards, >>> Benjamin >>> >> >> > No, PyPy has an RPython math module which calls libc. > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjhui at yahoo.com Wed Nov 16 19:03:28 2011 From: cjhui at yahoo.com (Charlie Hui) Date: Wed, 16 Nov 2011 10:03:28 -0800 (PST) Subject: [pypy-dev] Pure Python Math Library In-Reply-To: References: Message-ID: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> http://en.literateprograms.org/Logarithm_Function_(Python) ceil seems pretty easy to implement... pow? Integer only? --C ________________________________ From: Blaine To: pypy-dev at python.org Sent: Wednesday, November 16, 2011 9:21 AM Subject: [pypy-dev] Pure Python Math Library Does anyone know of a pure python math library? I've been playing around with berp, which is a python3 to haskell translator and compiler, and it works great as long as you don't go crazy with C extensions. It's highly experimental but fun to play around with. The only thing that I really miss is being able to use the math module. I asked the maintainer if it is possible to map into haskell's math library, but in the mean time a pure python math library would fit nicely since it would be compiled along with the rest of the python. I'm looking for log, log10, ceil, and pow mostly for my personal needs. It's funny now that things like pypy and berp exist. I find myself trying to locate pure python routines (like DFT) that would have no reason to exist with cpython, but make lots of sense with pypy. Anyway, just wondering if anyone had heard of such a thing. How do you even implement log? Thanks, Blaine _______________________________________________ pypy-dev mailing list pypy-dev at python.org http://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From frikker at gmail.com Wed Nov 16 20:01:38 2011 From: frikker at gmail.com (Blaine) Date: Wed, 16 Nov 2011 14:01:38 -0500 Subject: [pypy-dev] Pure Python Math Library In-Reply-To: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> References: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> Message-ID: Thanks Charlie, that's really helpful! Yeah ceil is easy. Pow can be done if log() and exp() are available, and exp() shouldn't be too hard I think. def pow(x,y): return exp(y*log(x)) Blaine On Wed, Nov 16, 2011 at 1:03 PM, Charlie Hui wrote: > http://en.literateprograms.org/Logarithm_Function_(Python) > ceil seems pretty easy to implement... > pow? Integer only? > > --C > > ------------------------------ > *From:* Blaine > *To:* pypy-dev at python.org > *Sent:* Wednesday, November 16, 2011 9:21 AM > *Subject:* [pypy-dev] Pure Python Math Library > > Does anyone know of a pure python math library? I've been playing around > with berp , which is a python3 to > haskell translator and compiler, and it works great as long as you don't go > crazy with C extensions. It's highly experimental but fun to play around > with. The only thing that I really miss is being able to use the math > module. I asked the maintainer if it is possible to map into haskell's math > library, but in the mean time a pure python math library would fit nicely > since it would be compiled along with the rest of the python. > > I'm looking for log, log10, ceil, and pow mostly for my personal needs. > > It's funny now that things like pypy and berp exist. I find myself trying > to locate pure python routines (like DFT) that would have no reason to > exist with cpython, but make lots of sense with pypy. > > Anyway, just wondering if anyone had heard of such a thing. How do you > even implement log? > > Thanks, > Blaine > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From estama at gmail.com Thu Nov 17 02:13:06 2011 From: estama at gmail.com (Elefterios Stamatogiannakis) Date: Thu, 17 Nov 2011 03:13:06 +0200 Subject: [pypy-dev] Slow sqlite user defined functions with pypy. In-Reply-To: References: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> Message-ID: <4EC45FA2.90409@gmail.com> The following code is a lot slower with pypy as compared to CPython. The code mainly measures the time taken to execute a simple SQLite user defined function (UDF) and a more complex one, 1000000 times each. Execution time for both queries is: CPython 2.7: 7 sec 489 msec Pypy nightly build: 28 sec 753 msec Execution time for simple query only is: CPython 2.7: 1 sec 39 msec Pypy nightly build: 13 sec 645 msec Pypy seems to not jit at all when a (pypy) Python function is called from C. Also based on the simple query times, pypy seems to have massive overhead (nearly 13 times slower) when executing callbacks from C code. lefteris. --- code -- import sqlite3 import datetime # Simple function def testfun(*args): return 1 # Complex function def sectohuman(*args): secs=int(args[0]) h='' days=secs/86400 if days > 0: h+=str(days)+' day' if days > 1: h+='s' h+=' ' secs=secs % 86400 hours=secs/3600 if hours > 0: h+=str(hours)+' hour' if hours > 1: h+='s' h+=' ' secs=secs % 3600 mins=secs/60 if mins > 0: h+=str(mins)+' min ' secs=secs % 60 if secs > 0: h+=str(secs)+' sec' return h con=sqlite3.Connection('') con.create_function('testfun', -1, testfun) con.create_function('sectohuman', -1, sectohuman) cur=con.cursor() cur.execute('create table l(a);') cur.execute('begin;') for i in xrange(1000000): cur.execute('insert into l values(?)',(i,)) before=datetime.datetime.now() # Simple query a=list(cur.execute('select sum(testfun(a)) as s from l')) # Complex query a=list(cur.execute('select sum(length(sectohuman(a))) as s from l')) after=datetime.datetime.now() tmdiff=after-before print "Execution time is %s min. %s sec %s msec" %((int(tmdiff.days)*24*60+(int(tmdiff.seconds)/60),(int(tmdiff.seconds)%60),(int(tmdiff.microseconds)/1000))) From william.leslie.ttg at gmail.com Thu Nov 17 02:24:27 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Thu, 17 Nov 2011 12:24:27 +1100 Subject: [pypy-dev] Slow sqlite user defined functions with pypy. In-Reply-To: References: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> <4EC45FA2.90409@gmail.com> Message-ID: Ack. On 17 November 2011 12:23, William ML Leslie wrote: > On 17 November 2011 12:13, Elefterios Stamatogiannakis wrote: >> Pypy seems to not jit at all when a (pypy) Python function is called from C. > > Calls to native functions must be residualised, as there is no way to > tell what state gives rise to the call to the UDF. > > If there was a loop inside the UDF, that would still be compiled. ?But > the loop that occurs in the query must just call the C function in the > usual way, as the JIT has no idea what it might do. -- William Leslie From alex.gaynor at gmail.com Thu Nov 17 02:56:45 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 16 Nov 2011 20:56:45 -0500 Subject: [pypy-dev] Slow sqlite user defined functions with pypy. In-Reply-To: References: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> <4EC45FA2.90409@gmail.com> Message-ID: On Wed, Nov 16, 2011 at 8:24 PM, William ML Leslie < william.leslie.ttg at gmail.com> wrote: > Ack. > > On 17 November 2011 12:23, William ML Leslie > wrote: > > On 17 November 2011 12:13, Elefterios Stamatogiannakis > wrote: > >> Pypy seems to not jit at all when a (pypy) Python function is called > from C. > > > > Calls to native functions must be residualised, as there is no way to > > tell what state gives rise to the call to the UDF. > > > > If there was a loop inside the UDF, that would still be compiled. But > > the loop that occurs in the query must just call the C function in the > > usual way, as the JIT has no idea what it might do. > > -- > William Leslie > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > The JIT compiles functions without loops too now, so this should be jitted. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Thu Nov 17 09:42:28 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 17 Nov 2011 09:42:28 +0100 Subject: [pypy-dev] Slow sqlite user defined functions with pypy. In-Reply-To: References: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> <4EC45FA2.90409@gmail.com> Message-ID: <4EC4C8F4.2070706@gmail.com> On 11/17/2011 02:56 AM, Alex Gaynor wrote: > The JIT compiles functions without loops too now, so this should be jitted. ctypes callbacks still go through the old _rawffi, so it's possible that this introduces some unneeded overhead. ciao, Anto From estama at gmail.com Thu Nov 17 13:02:35 2011 From: estama at gmail.com (Elefterios Stamatogiannakis) Date: Thu, 17 Nov 2011 14:02:35 +0200 Subject: [pypy-dev] Slow sqlite user defined functions with pypy. In-Reply-To: <4EC4C8F4.2070706@gmail.com> References: <1321466608.67848.YahooMailNeo@web160713.mail.bf1.yahoo.com> <4EC45FA2.90409@gmail.com> <4EC4C8F4.2070706@gmail.com> Message-ID: <4EC4F7DB.4000705@gmail.com> Thanks to all for your answers. I took some time to think some more about the results and: For the simple function (which returns 1), CPython roughly takes 1 sec and pypy 13 secs. IMHO, this case reveals pypy's callback overhead. For the complex function case, CPython roughly takes 6 secs and pypy 15 secs. If we assume that the overhead will be the same as in the simple function case, and subtract it, then CPython roughly took 5 secs to execute the complex function and pypy 2 secs. From the above, i think that my jit guess was wrong (as Alex said), and jit indeed works, but the overhead of the pypy callbacks is indeed quite large. I know that callbacks from C code aren't so frequent in Python programs, but my project: http://code.google.com/p/madis/ lives on them. So i would be glad if it could be solved (and madis's data processing would become a lot faster). Alternatively, is there something that i could do in my code that would overcome this overhead? Like signalling to the jit engine that this function should be always jitted and to do not waste time updating statistics (or other tracking information) on it? thanks lefteris. On 17/11/11 10:42, Antonio Cuni wrote: > On 11/17/2011 02:56 AM, Alex Gaynor wrote: > >> The JIT compiles functions without loops too now, so this should be jitted. > > ctypes callbacks still go through the old _rawffi, so it's possible that this > introduces some unneeded overhead. > > ciao, > Anto > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From fijall at gmail.com Fri Nov 18 11:01:42 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 18 Nov 2011 12:01:42 +0200 Subject: [pypy-dev] New pypy website Message-ID: Hello everyone Ante Salinovic made us a great website for the upcoming 1.7 PyPy release. I would like to see some feedback on it. The preliminary version is now available at http://pypy-site.ep.io Among other things it's missing some more success stories as well as donate buttons for numpy and py3k. Cheers, fijal From jacob at openend.se Fri Nov 18 12:13:25 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Fri, 18 Nov 2011 12:13:25 +0100 Subject: [pypy-dev] New pypy website In-Reply-To: References: Message-ID: <201111181213.34331.jacob@openend.se> Friday 18 November 2011 you wrote: > Hello everyone > > Ante Salinovic made us a great website for the upcoming 1.7 PyPy > release. I would like to see some feedback on it. The preliminary > version is now available at http://pypy-site.ep.io Among other things > it's missing some more success stories as well as donate buttons for > numpy and py3k. The user interface on the website is seriously muddled, with the links by each paragraph changing the contents of that paragraph. It means that you can't view the whole topic at the same time, straining your short term memory. It is in short really horrible usability. I also dislike the colour scheme, which does not connect to anything else we have. Use the blue colour from the blog instead. Jacob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1954 bytes Desc: not available URL: From holger at merlinux.eu Fri Nov 18 12:17:44 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 18 Nov 2011 11:17:44 +0000 Subject: [pypy-dev] New pypy website In-Reply-To: References: Message-ID: <20111118111744.GG27920@merlinux.eu> Hi Maciej, On Fri, Nov 18, 2011 at 12:01 +0200, Maciej Fijalkowski wrote: > Hello everyone > > Ante Salinovic made us a great website for the upcoming 1.7 PyPy > release. I would like to see some feedback on it. The preliminary > version is now available at http://pypy-site.ep.io Among other things > it's missing some more success stories as well as donate buttons for > numpy and py3k. i like it, Kudos to Ante Salinovic! Where are the raw texts kept? I looked into the new pypy-site repository and only found code and html/css. holger From arkanosis at gmail.com Fri Nov 18 12:22:40 2011 From: arkanosis at gmail.com (=?UTF-8?B?SsOpcsOpbWllIFJvcXVldA==?=) Date: Fri, 18 Nov 2011 12:22:40 +0100 Subject: [pypy-dev] New pypy website In-Reply-To: References: Message-ID: Hello, 2011/11/18 Maciej Fijalkowski : > Ante Salinovic made us a great website for the upcoming 1.7 PyPy > release. I would like to see some feedback on it. The preliminary > version is now available at http://pypy-site.ep.io Among other things > it's missing some more success stories as well as donate buttons for > numpy and py3k. Very nice, great work! May I suggest to make the moving box with the architecture next to the big ? Download PyPy ? button clickable as well? And to add a link to the ? other platforms ? next to it? I know there is another link for all downloads at the top of the page, but it's scattered. Also, it's not clear that the search box for compatibility is an input field and what it is there for (but the dynamic filtering works very well). Last thing for the UI, syntax highlighting in the code snippets would be an awesome plus! Now, for the content, I think it's missing a small chart on the front page to illustrate the speed improvements over CPython. Again, well done. Best regards, -- J?r?mie From santagada at gmail.com Fri Nov 18 12:22:57 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Fri, 18 Nov 2011 09:22:57 -0200 Subject: [pypy-dev] New pypy website In-Reply-To: <201111181213.34331.jacob@openend.se> References: <201111181213.34331.jacob@openend.se> Message-ID: On Fri, Nov 18, 2011 at 9:13 AM, Jacob Hall?n wrote: > Friday 18 November 2011 you wrote: >> Hello everyone >> >> Ante Salinovic made us a great website for the upcoming 1.7 PyPy >> release. I would like to see some feedback on it. The preliminary >> version is now available at http://pypy-site.ep.io Among other things >> it's missing some more success stories as well as donate buttons for >> numpy and py3k. > > > The user interface on the website is seriously muddled, with the links by each > paragraph changing the contents of that paragraph. > > It means that you can't view the whole topic at the same time, straining your > short term memory. It is in short really horrible usability. Maybe it could just jump around on the page like many modern sites are doing. > I also dislike the colour scheme, which does not connect to anything else we > have. Use the blue colour from the blog instead. I really liked the colors and the layout. For success stories I think maybe myhdl can be there... I don't know any more. -- Leonardo Santagada From arkanosis at gmail.com Fri Nov 18 12:26:57 2011 From: arkanosis at gmail.com (=?UTF-8?B?SsOpcsOpbWllIFJvcXVldA==?=) Date: Fri, 18 Nov 2011 12:26:57 +0100 Subject: [pypy-dev] New pypy website In-Reply-To: <201111181213.34331.jacob@openend.se> References: <201111181213.34331.jacob@openend.se> Message-ID: 2011/11/18 Jacob Hall?n : > The user interface on the website is seriously muddled, with the links by each > paragraph changing the contents of that paragraph. > > It means that you can't view the whole topic at the same time, straining your > short term memory. It is in short really horrible usability. Ah, I have to admit I hadn't even realized there was more text when clicking on these links. Yes, this is not very usable. -- J?r?mie From arigo at tunes.org Fri Nov 18 12:49:45 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 18 Nov 2011 12:49:45 +0100 Subject: [pypy-dev] New pypy website In-Reply-To: References: <201111181213.34331.jacob@openend.se> Message-ID: Hi, I don't really care about the website design, and I will generally tend to prefer what I'm used to over something new. But the fact is that I really prefer *a lot* the existing pypy.org over the new one. It is html-as-expected: pages with reasonably-sized text that contain reasonably-colored links, all with a minimal clean design. (We can argue over the content, which needs a bit of clean-up, as usual.) The new website, on the other hand, is definitely of the lots-and-lots-of-magic style. I dislike the approach of putting all information *somehow* accessible by opening some sub-tab by clicking in some clever place in the middle of the page. How can we even have a url that directly references the installation instructions? So -1 from me. I also fail to see the point of listing all twitter entries that use the word "pypy", because it's usually unrelated. A bient?t, Armin. From bradskins at gmail.com Fri Nov 18 23:04:48 2011 From: bradskins at gmail.com (Bradley Saulteaux) Date: Fri, 18 Nov 2011 22:04:48 +0000 (UTC) Subject: [pypy-dev] Building pypy on freebsd 64bit References: <4BE9455E.9070600@yandex-team.ru> <4BE948D5.1080700@yandex-team.ru> Message-ID: Hello, You can find a port at: https://github.com/DragonSA/pypy Works for me. From fijall at gmail.com Mon Nov 21 11:35:51 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 21 Nov 2011 12:35:51 +0200 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot Message-ID: ================================== PyPy 1.7 - widening the sweet spot ================================== We're pleased to announce the 1.7 release of PyPy. As became a habit, this release brings a lot of bugfixes and performance improvements over the 1.6 release. However, unlike the previous releases, the focus has been on widening the "sweet spot" of PyPy. That is, classes of Python code that PyPy can greatly speed up should be vastly improved with this release. You can download the 1.7 release here: http://pypy.org/download.html What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (`pypy 1.7 and cpython 2.7.1`_ performance comparison) due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or Windows 32. Windows 64 work is ongoing, but not yet natively supported. The main topic of this release is widening the range of code which PyPy can greatly speed up. On average on our benchmark suite, PyPy 1.7 is around **30%** faster than PyPy 1.6 and up to **20 times** faster on some benchmarks. .. _`pypy 1.7 and cpython 2.7.1`: http://speed.pypy.org Highlights ========== * Numerous performance improvements. There are too many examples which python constructs now should behave faster to list them. * Bugfixes and compatibility fixes with CPython. * Windows fixes. * PyPy now comes with stackless features enabled by default. However, any loop using stackless features will interrupt the JIT for now, so no real performance improvement for stackless-based programs. Contact pypy-dev for info how to help on removing this restriction. * NumPy effort in PyPy was renamed numpypy. In order to try using it, simply write:: import numpypy as numpy at the beginning of your program. There is a huge progress on numpy in PyPy since 1.6, the main feature being implementation of dtypes. * JSON encoder (but not decoder) has been replaced with a new one. This one is written in pure Python, but is known to outperform CPython's C extension up to **2 times** in some cases. It's about **20 times** faster than the one that we had in 1.6. * The memory footprint of some of our RPython modules has been drastically improved. This should impact any applications using for example cryptography, like tornado. * There was some progress in exposing even more CPython C API via cpyext. Things that didn't make it, expect in 1.8 soon ============================================== There is an ongoing work, which while didn't make it to the release, is probably worth mentioning here. This is what you should probably expect in 1.8 some time soon: * Specialized list implementation. There is a branch that implements lists of integers/floats/strings as compactly as array.array. This should drastically improve performance/memory impact of some applications * NumPy effort is progressing forward, with multi-dimensional arrays coming soon. * There are two brand new JIT assembler backends, notably for the PowerPC and ARM processors. Fundraising =========== It's maybe worth mentioning that we're running fundraising campaigns for NumPy effort in PyPy and for Python 3 in PyPy. In case you want to see any of those happen faster, we urge you to donate to `numpy proposal`_ or `py3k proposal`_. In case you want PyPy to progress, but you trust us with the general direction, you can always donate to the `general pot`_. .. _`numpy proposal`: http://pypy.org/numpydonate.html .. _`py3k proposal`: http://pypy.org/py3donate.html .. _`general pot`: http://pypy.org From phyo.arkarlwin at gmail.com Mon Nov 21 14:33:42 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Mon, 21 Nov 2011 20:03:42 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: OK here is the link if everyone wondering where to download : https://bitbucket.org/pypy/pypy/downloads/pypy-1.7-linux64.tar.bz2 On Mon, Nov 21, 2011 at 5:05 PM, Maciej Fijalkowski wrote: > ================================== > PyPy 1.7 - widening the sweet spot > ================================== > > We're pleased to announce the 1.7 release of PyPy. As became a habit, this > release brings a lot of bugfixes and performance improvements over the 1.6 > release. However, unlike the previous releases, the focus has been on > widening > the "sweet spot" of PyPy. That is, classes of Python code that PyPy can > greatly > speed up should be vastly improved with this release. You can download the > 1.7 > release here: > > http://pypy.org/download.html > > What is PyPy? > ============= > > PyPy is a very compliant Python interpreter, almost a drop-in replacement > for > CPython 2.7. It's fast (`pypy 1.7 and cpython 2.7.1`_ performance > comparison) > due to its integrated tracing JIT compiler. > > This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or > Windows 32. Windows 64 work is ongoing, but not yet natively supported. > > The main topic of this release is widening the range of code which PyPy > can greatly speed up. On average on > our benchmark suite, PyPy 1.7 is around **30%** faster than PyPy 1.6 and up > to **20 times** faster on some benchmarks. > > .. _`pypy 1.7 and cpython 2.7.1`: http://speed.pypy.org > > > Highlights > ========== > > * Numerous performance improvements. There are too many examples which > python > constructs now should behave faster to list them. > > * Bugfixes and compatibility fixes with CPython. > > * Windows fixes. > > * PyPy now comes with stackless features enabled by default. However, > any loop using stackless features will interrupt the JIT for now, so no > real > performance improvement for stackless-based programs. Contact pypy-dev for > info how to help on removing this restriction. > > * NumPy effort in PyPy was renamed numpypy. In order to try using it, > simply > write:: > > import numpypy as numpy > > at the beginning of your program. There is a huge progress on numpy in > PyPy > since 1.6, the main feature being implementation of dtypes. > > * JSON encoder (but not decoder) has been replaced with a new one. This one > is written in pure Python, but is known to outperform CPython's C > extension > up to **2 times** in some cases. It's about **20 times** faster than > the one that we had in 1.6. > > * The memory footprint of some of our RPython modules has been drastically > improved. This should impact any applications using for example > cryptography, > like tornado. > > * There was some progress in exposing even more CPython C API via cpyext. > > Things that didn't make it, expect in 1.8 soon > ============================================== > > There is an ongoing work, which while didn't make it to the release, is > probably worth mentioning here. This is what you should probably expect in > 1.8 some time soon: > > * Specialized list implementation. There is a branch that implements lists > of > integers/floats/strings as compactly as array.array. This should > drastically > improve performance/memory impact of some applications > > * NumPy effort is progressing forward, with multi-dimensional arrays coming > soon. > > * There are two brand new JIT assembler backends, notably for the PowerPC > and > ARM processors. > > Fundraising > =========== > > It's maybe worth mentioning that we're running fundraising campaigns for > NumPy effort in PyPy and for Python 3 in PyPy. In case you want to see any > of those happen faster, we urge you to donate to `numpy proposal`_ or > `py3k proposal`_. In case you want PyPy to progress, but you trust us with > the general direction, you can always donate to the `general pot`_. > > .. _`numpy proposal`: http://pypy.org/numpydonate.html > .. _`py3k proposal`: http://pypy.org/py3donate.html > .. _`general pot`: http://pypy.org > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phyo.arkarlwin at gmail.com Mon Nov 21 15:08:16 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Mon, 21 Nov 2011 20:38:16 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: https://bitbucket.org/pypy/pypy/downloads/pypy-1.7-win32-c.zip can u try?> On Mon, Nov 21, 2011 at 5:05 PM, Maciej Fijalkowski wrote: > ================================== > PyPy 1.7 - widening the sweet spot > ================================== > > We're pleased to announce the 1.7 release of PyPy. As became a habit, this > release brings a lot of bugfixes and performance improvements over the 1.6 > release. However, unlike the previous releases, the focus has been on > widening > the "sweet spot" of PyPy. That is, classes of Python code that PyPy can > greatly > speed up should be vastly improved with this release. You can download the > 1.7 > release here: > > http://pypy.org/download.html > > What is PyPy? > ============= > > PyPy is a very compliant Python interpreter, almost a drop-in replacement > for > CPython 2.7. It's fast (`pypy 1.7 and cpython 2.7.1`_ performance > comparison) > due to its integrated tracing JIT compiler. > > This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or > Windows 32. Windows 64 work is ongoing, but not yet natively supported. > > The main topic of this release is widening the range of code which PyPy > can greatly speed up. On average on > our benchmark suite, PyPy 1.7 is around **30%** faster than PyPy 1.6 and up > to **20 times** faster on some benchmarks. > > .. _`pypy 1.7 and cpython 2.7.1`: http://speed.pypy.org > > > Highlights > ========== > > * Numerous performance improvements. There are too many examples which > python > constructs now should behave faster to list them. > > * Bugfixes and compatibility fixes with CPython. > > * Windows fixes. > > * PyPy now comes with stackless features enabled by default. However, > any loop using stackless features will interrupt the JIT for now, so no > real > performance improvement for stackless-based programs. Contact pypy-dev for > info how to help on removing this restriction. > > * NumPy effort in PyPy was renamed numpypy. In order to try using it, > simply > write:: > > import numpypy as numpy > > at the beginning of your program. There is a huge progress on numpy in > PyPy > since 1.6, the main feature being implementation of dtypes. > > * JSON encoder (but not decoder) has been replaced with a new one. This one > is written in pure Python, but is known to outperform CPython's C > extension > up to **2 times** in some cases. It's about **20 times** faster than > the one that we had in 1.6. > > * The memory footprint of some of our RPython modules has been drastically > improved. This should impact any applications using for example > cryptography, > like tornado. > > * There was some progress in exposing even more CPython C API via cpyext. > > Things that didn't make it, expect in 1.8 soon > ============================================== > > There is an ongoing work, which while didn't make it to the release, is > probably worth mentioning here. This is what you should probably expect in > 1.8 some time soon: > > * Specialized list implementation. There is a branch that implements lists > of > integers/floats/strings as compactly as array.array. This should > drastically > improve performance/memory impact of some applications > > * NumPy effort is progressing forward, with multi-dimensional arrays coming > soon. > > * There are two brand new JIT assembler backends, notably for the PowerPC > and > ARM processors. > > Fundraising > =========== > > It's maybe worth mentioning that we're running fundraising campaigns for > NumPy effort in PyPy and for Python 3 in PyPy. In case you want to see any > of those happen faster, we urge you to donate to `numpy proposal`_ or > `py3k proposal`_. In case you want PyPy to progress, but you trust us with > the general direction, you can always donate to the `general pot`_. > > .. _`numpy proposal`: http://pypy.org/numpydonate.html > .. _`py3k proposal`: http://pypy.org/py3donate.html > .. _`general pot`: http://pypy.org > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Mon Nov 21 15:14:45 2011 From: brian at python.org (Brian Curtin) Date: Mon, 21 Nov 2011 08:14:45 -0600 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: On Mon, Nov 21, 2011 at 08:08, Phyo Arkar wrote: > https://bitbucket.org/pypy/pypy/downloads/pypy-1.7-win32-c.zip > can u try?> I'd try it if it existed :) From phyo.arkarlwin at gmail.com Mon Nov 21 15:32:49 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Mon, 21 Nov 2011 21:02:49 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: https://bitbucket.org/pypy/pypy/downloads/ true , no Windows in existance. On Mon, Nov 21, 2011 at 8:44 PM, Brian Curtin wrote: > On Mon, Nov 21, 2011 at 08:08, Phyo Arkar > wrote: > > https://bitbucket.org/pypy/pypy/downloads/pypy-1.7-win32-c.zip > > can u try?> > > I'd try it if it existed :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From russel at russel.org.uk Mon Nov 21 17:14:07 2011 From: russel at russel.org.uk (Russel Winder) Date: Mon, 21 Nov 2011 16:14:07 +0000 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: <1321892047.16283.13.camel@anglides.russel.org.uk> On Mon, 2011-11-21 at 20:03 +0630, Phyo Arkar wrote: > OK here is the link if everyone wondering where to download : > > > > https://bitbucket.org/pypy/pypy/downloads/pypy-1.7-linux64.tar.bz2 > This build has been linked against libssl0.9.8 and so will not run on vanilla Debian Unstable. The workaround is to manually download and install the libssl0.9.8 package from the Debian Testing repository. Irritating, but works. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder at ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel at russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From phyo.arkarlwin at gmail.com Mon Nov 21 17:21:04 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Mon, 21 Nov 2011 22:51:04 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: <1321892047.16283.13.camel@anglides.russel.org.uk> References: <1321892047.16283.13.camel@anglides.russel.org.uk> Message-ID: Dont install 0.9.8 , two SSL lib will make everything confused. , make a new link , directly to 1.0.0 so file as 0.9.8 On Mon, Nov 21, 2011 at 10:44 PM, Russel Winder wrote: > On Mon, 2011-11-21 at 20:03 +0630, Phyo Arkar wrote: > > OK here is the link if everyone wondering where to download : > > > > > > > > https://bitbucket.org/pypy/pypy/downloads/pypy-1.7-linux64.tar.bz2 > > > > This build has been linked against libssl0.9.8 and so will not run on > vanilla Debian Unstable. The workaround is to manually download and > install the libssl0.9.8 package from the Debian Testing repository. > Irritating, but works. > > > -- > Russel. > > ============================================================================= > Dr Russel Winder t: +44 20 7585 2200 voip: > sip:russel.winder at ekiga.net > 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel at russel.org.uk > London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder > -------------- next part -------------- An HTML attachment was scrubbed... URL: From naylor.b.david at gmail.com Mon Nov 21 17:30:31 2011 From: naylor.b.david at gmail.com (David Naylor) Date: Mon, 21 Nov 2011 18:30:31 +0200 Subject: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM Message-ID: <201111211830.36212.naylor.b.david@gmail.com> Hi While trying to translate pypy-1.7 with the sandbox option, using pypy-1.6 I saw the translating pypy consume 13.3GB of memory (column Res in top). This is under FreeBSD-9RC2/amd64. I've never noticed this memory consumtion when translating pypy-1.6 with the stackless option. Is this perhaps a bug with pypy-1.6 or the source code for pypy-1.7? I'll try again using pypy-1.7 to do the translating once it has been translated (I needed to restart since I was running the translations in parallel). Regards -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: This is a digitally signed message part. URL: From russel at russel.org.uk Mon Nov 21 17:33:45 2011 From: russel at russel.org.uk (Russel Winder) Date: Mon, 21 Nov 2011 16:33:45 +0000 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> Message-ID: <1321893225.16283.24.camel@anglides.russel.org.uk> On Mon, 2011-11-21 at 22:51 +0630, Phyo Arkar wrote: > Dont install 0.9.8 , two SSL lib will make everything confused. , make > a new link , directly to 1.0.0 so file as 0.9.8 > Why will things be confused? Debian is generally very careful about how the packages install exactly to avoid confusion between to soname variants of the same library. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder at ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel at russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From angelflow at yahoo.com Mon Nov 21 17:51:53 2011 From: angelflow at yahoo.com (Andy) Date: Mon, 21 Nov 2011 08:51:53 -0800 (PST) Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: <1321894313.66576.YahooMailNeo@web111308.mail.gq1.yahoo.com> Hi, Congrats on 1.7! >?any loop using stackless features will interrupt the JIT for now, so no real> performance improvement for stackless-based programs. I have a program that runs on gevent. Does this mean PyPy JIT wouldn't work with it? If so, will JIT work in future release? > JSON encoder (but not decoder) has been replaced with a new one. ?? So JSON decoding in PyPy is still slower than in C-extension? Will there be new decoder to fix that? Thanks and keep up the great work. Andy ________________________________ From: Maciej Fijalkowski To: PyPy Developer Mailing List Sent: Monday, November 21, 2011 5:35 AM Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot ================================== PyPy 1.7 - widening the sweet spot ================================== We're pleased to announce the 1.7 release of PyPy. As became a habit, this release brings a lot of bugfixes and performance improvements over the 1.6 release. However, unlike the previous releases, the focus has been on widening the "sweet spot" of PyPy. That is, classes of Python code that PyPy can greatly speed up should be vastly improved with this release. You can download the 1.7 release here: ? ? http://pypy.org/download.html What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (`pypy 1.7 and cpython 2.7.1`_ performance comparison) due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or Windows 32. Windows 64 work is ongoing, but not yet natively supported. The main topic of this release is widening the range of code which PyPy can greatly speed up. On average on our benchmark suite, PyPy 1.7 is around **30%** faster than PyPy 1.6 and up to **20 times** faster on some benchmarks. .. _`pypy 1.7 and cpython 2.7.1`: http://speed.pypy.org Highlights ========== * Numerous performance improvements. There are too many examples which python ? constructs now should behave faster to list them. * Bugfixes and compatibility fixes with CPython. * Windows fixes. * PyPy now comes with stackless features enabled by default. However, ? any loop using stackless features will interrupt the JIT for now, so no real ? performance improvement for stackless-based programs. Contact pypy-dev for ? info how to help on removing this restriction. * NumPy effort in PyPy was renamed numpypy. In order to try using it, simply ? write:: ? ? import numpypy as numpy ? at the beginning of your program. There is a huge progress on numpy in PyPy ? since 1.6, the main feature being implementation of dtypes. * JSON encoder (but not decoder) has been replaced with a new one. This one ? is written in pure Python, but is known to outperform CPython's C extension ? up to **2 times** in some cases. It's about **20 times** faster than ? the one that we had in 1.6. * The memory footprint of some of our RPython modules has been drastically ? improved. This should impact any applications using for example cryptography, ? like tornado. * There was some progress in exposing even more CPython C API via cpyext. Things that didn't make it, expect in 1.8 soon ============================================== There is an ongoing work, which while didn't make it to the release, is probably worth mentioning here. This is what you should probably expect in 1.8 some time soon: * Specialized list implementation. There is a branch that implements lists of ? integers/floats/strings as compactly as array.array. This should drastically ? improve performance/memory impact of some applications * NumPy effort is progressing forward, with multi-dimensional arrays coming ? soon. * There are two brand new JIT assembler backends, notably for the PowerPC and ? ARM processors. Fundraising =========== It's maybe worth mentioning that we're running fundraising campaigns for NumPy effort in PyPy and for Python 3 in PyPy. In case you want to see any of those happen faster, we urge you to donate to `numpy proposal`_ or `py3k proposal`_. In case you want PyPy to progress, but you trust us with the general direction, you can always donate to the `general pot`_. .. _`numpy proposal`: http://pypy.org/numpydonate.html .. _`py3k proposal`: http://pypy.org/py3donate.html .. _`general pot`: http://pypy.org _______________________________________________ pypy-dev mailing list pypy-dev at python.org http://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From naylor.b.david at gmail.com Mon Nov 21 17:55:27 2011 From: naylor.b.david at gmail.com (David Naylor) Date: Mon, 21 Nov 2011 18:55:27 +0200 Subject: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM In-Reply-To: <201111211830.36212.naylor.b.david@gmail.com> References: <201111211830.36212.naylor.b.david@gmail.com> Message-ID: <201111211855.39558.naylor.b.david@gmail.com> On Monday, 21 November 2011 18:30:31 David Naylor wrote: > Hi > > While trying to translate pypy-1.7 with the sandbox option, using pypy-1.6 > I saw the translating pypy consume 13.3GB of memory (column Res in top). > This is under FreeBSD-9RC2/amd64. This isn't specific to the sandbox option. The default options used also cause this memory usage. On a box with 16GB of RAM the translation was causing extensive swapping (I estimate a peak memory of ~15GB). This seems to happen during jitcodewriter. > Is this perhaps a bug with pypy-1.6 or the source code for pypy-1.7? I'll > try again using pypy-1.7 to do the translating once it has been translated > (I needed to restart since I was running the translations in parallel). I will also run a comparitive build with python-2.7 and report back... -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: This is a digitally signed message part. URL: From lac at openend.se Mon Nov 21 19:16:49 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 21 Nov 2011 19:16:49 +0100 Subject: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM In-Reply-To: Message from David Naylor of "Mon, 21 Nov 2011 18:55:27 +0200." <201111211855.39558.naylor.b.david@gmail.com> References: <201111211830.36212.naylor.b.david@gmail.com><201111211855.39558.naylor.b.david@gmail.com> Message-ID: <201111211816.pALIGnC3021202@theraft.openend.se> Are you using gcc 4.2? If so , you have this bug. https://bugs.launchpad.net/ubuntu/+source/gcc-4.2/+bug/187391 and you need a newer gcc. (Or maybe an older one would work, too, but this one doesn't). Laura From fijall at gmail.com Mon Nov 21 21:51:14 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 21 Nov 2011 22:51:14 +0200 Subject: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM In-Reply-To: <201111211816.pALIGnC3021202@theraft.openend.se> References: <201111211830.36212.naylor.b.david@gmail.com> <201111211855.39558.naylor.b.david@gmail.com> <201111211816.pALIGnC3021202@theraft.openend.se> Message-ID: On Mon, Nov 21, 2011 at 8:16 PM, Laura Creighton wrote: > > Are you using gcc 4.2? > > If so , you have this bug. ?https://bugs.launchpad.net/ubuntu/+source/gcc-4.2/+bug/187391 > > and you ?need a newer gcc. ?(Or maybe an older one would work, too, but this > one doesn't). > > Laura This would be before GCC can run. It's probably worth investigating, it surely should not use that much. PS. Is it possible to access a BSD box where I can try? If not I can probably provide you with some debugging tools. Cheers, fijal From holger at merlinux.eu Tue Nov 22 00:23:13 2011 From: holger at merlinux.eu (holger krekel) Date: Mon, 21 Nov 2011 23:23:13 +0000 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: <20111121232313.GI27920@merlinux.eu> On Mon, Nov 21, 2011 at 12:35 +0200, Maciej Fijalkowski wrote: > * PyPy now comes with stackless features enabled by default. However, > any loop using stackless features will interrupt the JIT for now, so no real > performance improvement for stackless-based programs. Contact pypy-dev for > info how to help on removing this restriction. Yes, please, could you talk a bit more explicitely about what is involved and what works/integrated and what doesn't? best, holger > * NumPy effort in PyPy was renamed numpypy. In order to try using it, simply > write:: > > import numpypy as numpy > > at the beginning of your program. There is a huge progress on numpy in PyPy > since 1.6, the main feature being implementation of dtypes. > > * JSON encoder (but not decoder) has been replaced with a new one. This one > is written in pure Python, but is known to outperform CPython's C extension > up to **2 times** in some cases. It's about **20 times** faster than > the one that we had in 1.6. > > * The memory footprint of some of our RPython modules has been drastically > improved. This should impact any applications using for example cryptography, > like tornado. > > * There was some progress in exposing even more CPython C API via cpyext. > > Things that didn't make it, expect in 1.8 soon > ============================================== > > There is an ongoing work, which while didn't make it to the release, is > probably worth mentioning here. This is what you should probably expect in > 1.8 some time soon: > > * Specialized list implementation. There is a branch that implements lists of > integers/floats/strings as compactly as array.array. This should drastically > improve performance/memory impact of some applications > > * NumPy effort is progressing forward, with multi-dimensional arrays coming > soon. > > * There are two brand new JIT assembler backends, notably for the PowerPC and > ARM processors. > > Fundraising > =========== > > It's maybe worth mentioning that we're running fundraising campaigns for > NumPy effort in PyPy and for Python 3 in PyPy. In case you want to see any > of those happen faster, we urge you to donate to `numpy proposal`_ or > `py3k proposal`_. In case you want PyPy to progress, but you trust us with > the general direction, you can always donate to the `general pot`_. > > .. _`numpy proposal`: http://pypy.org/numpydonate.html > .. _`py3k proposal`: http://pypy.org/py3donate.html > .. _`general pot`: http://pypy.org > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From drsalists at gmail.com Tue Nov 22 02:07:41 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 21 Nov 2011 17:07:41 -0800 Subject: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM In-Reply-To: <201111211830.36212.naylor.b.david@gmail.com> References: <201111211830.36212.naylor.b.david@gmail.com> Message-ID: On 11/21/11, David Naylor wrote: > Hi > > While trying to translate pypy-1.7 with the sandbox option, using pypy-1.6 I > saw the translating pypy consume 13.3GB of memory (column Res in top). This > is under FreeBSD-9RC2/amd64. > I also get a lot of memory use when building pypy 1.6 on a 64 bit Debian system with gcc 4.4.5. The machine has 3 Gigabytes of physmem. I let one build attempt run for 2 or 3 days before killing it. The same system built pypy OK with a 32 bit OS, though they were builds of nearby pypy trunk's, not 1.6. > I've never noticed this memory consumtion when translating pypy-1.6 with the > stackless option. > > Is this perhaps a bug with pypy-1.6 or the source code for pypy-1.7? I'll > try > again using pypy-1.7 to do the translating once it has been translated (I > needed to restart since I was running the translations in parallel). > > Regards > From drsalists at gmail.com Tue Nov 22 02:14:18 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 21 Nov 2011 17:14:18 -0800 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: Message-ID: Very nice. Passes my barrage of backshift automatic tests. I'm excited about investigating the performance benefits for my deduplication algorithm. From phyo.arkarlwin at gmail.com Tue Nov 22 06:58:13 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Tue, 22 Nov 2011 12:28:13 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: <1321893225.16283.24.camel@anglides.russel.org.uk> References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: >The workaround is to manually download and >install the libssl0.9.8 package from the Debian Testing repository. >Irritating, but works. Mixing repositories .... I have been in dependancy hell because of that. On Mon, Nov 21, 2011 at 11:03 PM, Russel Winder wrote: > On Mon, 2011-11-21 at 22:51 +0630, Phyo Arkar wrote: > > Dont install 0.9.8 , two SSL lib will make everything confused. , make > > a new link , directly to 1.0.0 so file as 0.9.8 > > > Why will things be confused? Debian is generally very careful about how > the packages install exactly to avoid confusion between to soname > variants of the same library. > > -- > Russel. > > ============================================================================= > Dr Russel Winder t: +44 20 7585 2200 voip: > sip:russel.winder at ekiga.net > 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel at russel.org.uk > London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 22 07:24:36 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 22 Nov 2011 08:24:36 +0200 Subject: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM In-Reply-To: References: <201111211830.36212.naylor.b.david@gmail.com> Message-ID: On Tue, Nov 22, 2011 at 3:07 AM, Dan Stromberg wrote: > On 11/21/11, David Naylor wrote: >> Hi >> >> While trying to translate pypy-1.7 with the sandbox option, using pypy-1.6 I >> saw the translating pypy consume 13.3GB of memory (column Res in top). ?This >> is under FreeBSD-9RC2/amd64. >> > > I also get a lot of memory use when building pypy 1.6 on a 64 bit > Debian system with gcc 4.4.5. > > The machine has 3 Gigabytes of physmem. ?I let one build attempt run > for 2 or 3 days before killing it. > > The same system built pypy OK with a 32 bit OS, though they were > builds of nearby pypy trunk's, not 1.6. 3G is borderline for 64bit. I build pypy on 3G, but it requires closing browsers etc. > >> I've never noticed this memory consumtion when translating pypy-1.6 with the >> stackless option. >> >> Is this perhaps a bug with pypy-1.6 or the source code for pypy-1.7? ?I'll >> try >> again using pypy-1.7 to do the translating once it has been translated (I >> needed to restart since I was running the translations in parallel). >> >> Regards >> > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From greg at quora.com Tue Nov 22 09:16:18 2011 From: greg at quora.com (Greg Price) Date: Tue, 22 Nov 2011 00:16:18 -0800 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: On Mon, Nov 21, 2011 at 9:58 PM, Phyo Arkar wrote: > >The workaround is to manually download and > >install the libssl0.9.8 package from the Debian Testing repository. > >Irritating, but works. > > Mixing repositories .... > I have been in dependancy hell because of that. Well, let's see. Rather than issue FUD from vague past experiences, shall we look at what dependencies the suggested package actually has? http://packages.debian.org/testing/libssl0.9.8 debconf, libc6, and zlib1g. I wouldn't worry about any of those having disappeared in sid. (And in fact they haven't.) Greg From phyo.arkarlwin at gmail.com Tue Nov 22 09:38:59 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Tue, 22 Nov 2011 15:08:59 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: in my Gentoo based Sabayon they are already gone. and i believe next latest Ubuntu may use 1.0.0 too. On Tue, Nov 22, 2011 at 2:46 PM, Greg Price wrote: > On Mon, Nov 21, 2011 at 9:58 PM, Phyo Arkar > wrote: > > >The workaround is to manually download and > > >install the libssl0.9.8 package from the Debian Testing repository. > > >Irritating, but works. > > > > Mixing repositories .... > > I have been in dependancy hell because of that. > > Well, let's see. Rather than issue FUD from vague past experiences, > shall we look at what dependencies the suggested package actually has? > > *http://packages.debian.org/testing/libssl0.9.8* > > > debconf, libc6, and zlib1g. I wouldn't worry about any of those having > disappeared in sid. (And in fact they haven't.) > > Greg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 22 09:47:43 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 22 Nov 2011 10:47:43 +0200 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: <20111121232313.GI27920@merlinux.eu> References: <20111121232313.GI27920@merlinux.eu> Message-ID: On Tue, Nov 22, 2011 at 1:23 AM, holger krekel wrote: > On Mon, Nov 21, 2011 at 12:35 +0200, Maciej Fijalkowski wrote: >> * PyPy now comes with stackless features enabled by default. However, >> ? any loop using stackless features will interrupt the JIT for now, so no real >> ? performance improvement for stackless-based programs. Contact pypy-dev for >> ? info how to help on removing this restriction. > > Yes, please, could you talk a bit more explicitely about what is involved > and what works/integrated and what doesn't? > > best, > holger > Everything works, but every instruction that can cause a switch (so pretty much using *any* of stackless features) will disable jitting for this loop. this is unideal, and I don't quite recall the exact reasons why, other than "more work is needed". Armin, can you elaborate? Cheers, fijal From arigo at tunes.org Tue Nov 22 09:51:35 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 22 Nov 2011 09:51:35 +0100 Subject: [pypy-dev] Translating pypy-sandbox eats 13GB of RAM In-Reply-To: <201111211855.39558.naylor.b.david@gmail.com> References: <201111211830.36212.naylor.b.david@gmail.com> <201111211855.39558.naylor.b.david@gmail.com> Message-ID: Hi, On Mon, Nov 21, 2011 at 17:55, David Naylor wrote: >> While trying to translate pypy-1.7 with the sandbox option, using pypy-1.6 >> I saw the translating pypy consume 13.3GB of memory (column Res in top). >> This is under FreeBSD-9RC2/amd64. > > This isn't specific to the sandbox option. ?The default options used also > cause this memory usage. PyPy used to contain a bug that causes infinite memory usage in one particular circumstance (literally -- so in practice, infinite swapping with no progress). It so happens that since about one month, translating a PyPy may trigger it. This means that in order to translate a PyPy, you need to run (either CPython or) a version of PyPy that contains the fix, which was done November 11. A bient?t, Armin. From arigo at tunes.org Tue Nov 22 10:10:45 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 22 Nov 2011 10:10:45 +0100 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: Hi Phyo, On Tue, Nov 22, 2011 at 09:38, Phyo Arkar wrote: > in my Gentoo based Sabayon they are already gone. and i believe next latest > Ubuntu may use 1.0.0 too. The point is of course that doing binary releases on Linux is always a bit of a problem, unless we ship everything as included static libraries, which is a bit nonsensical IMHO for a project like PyPy. Moreover, if openssl 1.0.0 is binary compatible with openssl 0.9.8, then I might rather put the blame on the openssl project for releasing two binary compatible versions with a different (un-cross-linkable) first number. But I will not do so: I guess that they are not supposed to be totally binary compatible, and there are subtle differences. Is there anyone that can (1) make sure the differences don't matter, possibly by looking at how the CPython source evolved; and (2) maybe come up with a way to say "link against openssl either 0.9.8 or 1.0.0" in the binary? A bient?t, Armin. From arigo at tunes.org Tue Nov 22 10:27:03 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 22 Nov 2011 10:27:03 +0100 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <20111121232313.GI27920@merlinux.eu> Message-ID: Hi, On Tue, Nov 22, 2011 at 09:47, Maciej Fijalkowski wrote: > Everything works, but every instruction that can cause a switch (so > pretty much using *any* of stackless features) will disable jitting > for this loop. this is unideal, and I don't quite recall the exact > reasons why, other than "more work is needed". Armin, can you > elaborate? It's actually only a switch() that disables JITting. The code for that is in pypy/module/_continuation/interp_continuation.py, workaround_disable_jit(). Read the comment. You may also try to comment out that function's body and rebuild pypy: you would get a pypy that is fast including on loops that use switch(), but that may crash if you use anywhere frame inspection (e.g. sys._getframe()). The fix needs to be done in the JIT, but it's not very clear how to fix it without an additional (small) overhead also in cases that don't use stackless at all. Anyway I suppose the first step is to do it in a branch, and worry about the small overhead later. Grep for FORCE_TOKEN in pypy/jit/metainterp: this is a resoperation that returns the address of the current assembler frame; in other words a pointer to the stack. But the stack may now move with _continuations. More precisely, the stacks are copied around in such a way that the currently running stack is always at the same location as it used to be; but the point is that when doing e.g. sys._getframe(bignum), we need to access a stack that may not be running at all right now. So we need a way to track such moves. For example it could be complemented with another operation (or some load of a global field) that would return some handle that changes when we switch(). The idea is that later, if and when we need to track down the saved stack, we can use the handle to know (maybe indirectly --- speed does not matter there) where the saved stack was copied to. A bient?t, Armin. From phyo.arkarlwin at gmail.com Tue Nov 22 10:36:16 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Tue, 22 Nov 2011 16:06:16 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: I understand the complication of binary release. i did cross reference 1.0.0 and it works (starts) pypy fine,. But i am not sure what will happen if something that uses crypting modules. may be release both versions of openssl (damn , python should have have that openssl dependancy in standard modules :( ) On Tue, Nov 22, 2011 at 3:40 PM, Armin Rigo wrote: > Hi Phyo, > > On Tue, Nov 22, 2011 at 09:38, Phyo Arkar > wrote: > > in my Gentoo based Sabayon they are already gone. and i believe next > latest > > Ubuntu may use 1.0.0 too. > > The point is of course that doing binary releases on Linux is always a > bit of a problem, unless we ship everything as included static > libraries, which is a bit nonsensical IMHO for a project like PyPy. > > Moreover, if openssl 1.0.0 is binary compatible with openssl 0.9.8, > then I might rather put the blame on the openssl project for releasing > two binary compatible versions with a different (un-cross-linkable) > first number. But I will not do so: I guess that they are not > supposed to be totally binary compatible, and there are subtle > differences. > > Is there anyone that can (1) make sure the differences don't matter, > possibly by looking at how the CPython source evolved; and (2) maybe > come up with a way to say "link against openssl either 0.9.8 or 1.0.0" > in the binary? > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nitralime at googlemail.com Tue Nov 22 14:15:56 2011 From: nitralime at googlemail.com (nitralime) Date: Tue, 22 Nov 2011 14:15:56 +0100 Subject: [pypy-dev] Building lxml Message-ID: <4ECBA08C.6090105@gmail.com> Hello! I'v tried to build lxml 2.3.2 with pypy 1.7 (linux64) and got the following error: ------------------------------------------------------------------------------------ $ pypy setup.py build_ext -i pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Building lxml version 2.3.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib running build_ext building 'lxml.etree' extension creating build creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/src creating build/temp.linux-x86_64-2.7/src/lxml cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/local/pypy/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w src/lxml/lxml.etree.c: In function ?__Pyx_PyObject_Pop?: src/lxml/lxml.etree.c:3683:16: error: ?PyListObject? has no member named ?allocated? src/lxml/lxml.etree.c:3688:30: error: ?PySet_Type? undeclared (first use in this function) src/lxml/lxml.etree.c:3688:30: note: each undeclared identifier is reported only once for each function it appears in src/lxml/lxml.etree.c: In function ?__pyx_f_4lxml_5etree_18_FileReaderContext__readDtd?: src/lxml/lxml.etree.c:72226:7: error: ?_save? undeclared (first use in this function) .... ... error: command 'cc' failed with exit status 1 ------------------------------------------------------------------------------------ Next I have tried the following (which is probabely a "nonsense"): ------------------------------------------------------------------------------------ $ pypy setup.py build_ext -i -I/usr/include/python2.7 pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Building lxml version 2.3.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib running build_ext building 'lxml.etree' extension cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 -I/usr/local/pypy/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so building 'lxml.objectify' extension cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 -I/usr/local/pypy/include -c src/lxml/lxml.objectify.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -w cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o /home/nik/tmp/lxml-2.3.2/src/lxml/objectify.pypy-17.so ------------------------------------------------------------------------------------ $ PYTHONPATH=src pypy pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:11) [PyPy 1.7.0 with GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``happy new year'' >>>> from lxml import etree Traceback (most recent call last): File "", line 1, in ImportError: unable to load extension module '/home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so': /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so: undefined symbol: PyClass_Type >>>> Has anyone ever tried to build lxml with pypy? What is the correct way to build an extension module? Any feedback is very much appreciated! Regards Nik PS: By the way, what do pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) mean? Both libraries are available on my PC. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 22 13:26:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 22 Nov 2011 14:26:49 +0200 Subject: [pypy-dev] Building lxml In-Reply-To: <4ECBA08C.6090105@gmail.com> References: <4ECBA08C.6090105@gmail.com> Message-ID: On Tue, Nov 22, 2011 at 3:15 PM, nitralime wrote: > Hello! > > I'v tried to build lxml 2.3.2 with pypy 1.7 (linux64) > and got the following error: > > ------------------------------------------------------------------------------------ > $ pypy setup.py build_ext -i > pypy: /usr/lib/libssl.so.0.9.8: no version information available (required > by pypy) > pypy: /usr/lib/libcrypto.so.0.9.8: no version information available > (required by pypy) > Building lxml version 2.3.2. > Building without Cython. > Using build configuration of libxslt 1.1.26 > Building against libxml2/libxslt in the following directory: /usr/lib > running build_ext > building 'lxml.etree' extension > creating build > creating build/temp.linux-x86_64-2.7 > creating build/temp.linux-x86_64-2.7/src > creating build/temp.linux-x86_64-2.7/src/lxml > cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/local/pypy/include -c > src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o > -w > src/lxml/lxml.etree.c: In function ?__Pyx_PyObject_Pop?: > src/lxml/lxml.etree.c:3683:16: error: ?PyListObject? has no member named > ?allocated? > src/lxml/lxml.etree.c:3688:30: error: ?PySet_Type? undeclared (first use in > this function) > src/lxml/lxml.etree.c:3688:30: note: each undeclared identifier is reported > only once for each function it appears in > src/lxml/lxml.etree.c: In function > ?__pyx_f_4lxml_5etree_18_FileReaderContext__readDtd?: > src/lxml/lxml.etree.c:72226:7: error: ?_save? undeclared (first use in this > function) > .... > ... > error: command 'cc' failed with exit status 1 > ------------------------------------------------------------------------------------ > > Next I have tried the following (which is probabely a "nonsense"): > > ------------------------------------------------------------------------------------ > $ pypy setup.py build_ext -i -I/usr/include/python2.7 > pypy: /usr/lib/libssl.so.0.9.8: no version information available (required > by pypy) > pypy: /usr/lib/libcrypto.so.0.9.8: no version information available > (required by pypy) > Building lxml version 2.3.2. > Building without Cython. > Using build configuration of libxslt 1.1.26 > Building against libxml2/libxslt in the following directory: /usr/lib > running build_ext > building 'lxml.etree' extension > cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 > -I/usr/local/pypy/include -c src/lxml/lxml.etree.c -o > build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w > cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -L/usr/lib > -lxslt -lexslt -lxml2 -lz -lm -o > /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so > building 'lxml.objectify' extension > cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 > -I/usr/local/pypy/include -c src/lxml/lxml.objectify.c -o > build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -w > cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -L/usr/lib > -lxslt -lexslt -lxml2 -lz -lm -o > /home/nik/tmp/lxml-2.3.2/src/lxml/objectify.pypy-17.so > ------------------------------------------------------------------------------------ > > $ PYTHONPATH=src pypy > pypy: /usr/lib/libssl.so.0.9.8: no version information available (required > by pypy) > pypy: /usr/lib/libcrypto.so.0.9.8: no version information available > (required by pypy) > Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:11) > [PyPy 1.7.0 with GCC 4.4.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > And now for something completely different: ``happy new year'' >>>>> from lxml import etree > Traceback (most recent call last): > ? File "", line 1, in > ImportError: unable to load extension module > '/home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so': > /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so: undefined symbol: > PyClass_Type >>>>> > > Has anyone ever tried to build lxml with pypy? > What is the correct way to build an extension module? > > Any feedback is very much appreciated! > > Regards > Nik > Hi. lxml is unsupported because it's a cython-generated extension and cython pokes inside internals of CPython objects. This is unsupported and likely will never be (pypy objects are different enough). The fix is to either fix cython or provide a different cython backend. There was some work on the latter, but it did not go very far. For now you have to assume lxml won't work on pypy unfortunately :-/ Cheers, fijal From brunogola at gmail.com Tue Nov 22 13:31:29 2011 From: brunogola at gmail.com (Bruno Gola) Date: Tue, 22 Nov 2011 10:31:29 -0200 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: Hi, Just to let you know I just uploaded the new release to the official PPA, so if you use Ubuntu you can install PyPy 1.7 using https://launchpad.net/~pypy/+archive/ppa Tests and feedback are welcome :-) Thanks, Bruno On Tue, Nov 22, 2011 at 7:36 AM, Phyo Arkar wrote: > I understand the complication of binary release. > > i did cross reference 1.0.0 and it works (starts) pypy fine,. But i am > not sure what will happen if something that uses crypting modules. > > may be release both versions of openssl (damn , python should have have > that openssl dependancy in standard modules :( ) > > > On Tue, Nov 22, 2011 at 3:40 PM, Armin Rigo wrote: > >> Hi Phyo, >> >> On Tue, Nov 22, 2011 at 09:38, Phyo Arkar >> wrote: >> > in my Gentoo based Sabayon they are already gone. and i believe next >> latest >> > Ubuntu may use 1.0.0 too. >> >> The point is of course that doing binary releases on Linux is always a >> bit of a problem, unless we ship everything as included static >> libraries, which is a bit nonsensical IMHO for a project like PyPy. >> >> Moreover, if openssl 1.0.0 is binary compatible with openssl 0.9.8, >> then I might rather put the blame on the openssl project for releasing >> two binary compatible versions with a different (un-cross-linkable) >> first number. But I will not do so: I guess that they are not >> supposed to be totally binary compatible, and there are subtle >> differences. >> >> Is there anyone that can (1) make sure the differences don't matter, >> possibly by looking at how the CPython source evolved; and (2) maybe >> come up with a way to say "link against openssl either 0.9.8 or 1.0.0" >> in the binary? >> >> >> A bient?t, >> >> Armin. >> > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phyo.arkarlwin at gmail.com Tue Nov 22 15:42:51 2011 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Tue, 22 Nov 2011 21:12:51 +0630 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: nice On Tue, Nov 22, 2011 at 7:01 PM, Bruno Gola wrote: > Hi, > > Just to let you know I just uploaded the new release to the official PPA, > so if you use Ubuntu you can install PyPy 1.7 using > https://launchpad.net/~pypy/+archive/ppa > > Tests and feedback are welcome :-) > > Thanks, > Bruno > > On Tue, Nov 22, 2011 at 7:36 AM, Phyo Arkar wrote: > >> I understand the complication of binary release. >> >> i did cross reference 1.0.0 and it works (starts) pypy fine,. But i am >> not sure what will happen if something that uses crypting modules. >> >> may be release both versions of openssl (damn , python should have have >> that openssl dependancy in standard modules :( ) >> >> >> On Tue, Nov 22, 2011 at 3:40 PM, Armin Rigo wrote: >> >>> Hi Phyo, >>> >>> On Tue, Nov 22, 2011 at 09:38, Phyo Arkar >>> wrote: >>> > in my Gentoo based Sabayon they are already gone. and i believe next >>> latest >>> > Ubuntu may use 1.0.0 too. >>> >>> The point is of course that doing binary releases on Linux is always a >>> bit of a problem, unless we ship everything as included static >>> libraries, which is a bit nonsensical IMHO for a project like PyPy. >>> >>> Moreover, if openssl 1.0.0 is binary compatible with openssl 0.9.8, >>> then I might rather put the blame on the openssl project for releasing >>> two binary compatible versions with a different (un-cross-linkable) >>> first number. But I will not do so: I guess that they are not >>> supposed to be totally binary compatible, and there are subtle >>> differences. >>> >>> Is there anyone that can (1) make sure the differences don't matter, >>> possibly by looking at how the CPython source evolved; and (2) maybe >>> come up with a way to say "link against openssl either 0.9.8 or 1.0.0" >>> in the binary? >>> >>> >>> A bient?t, >>> >>> Armin. >>> >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From naylor.b.david at gmail.com Tue Nov 22 20:39:46 2011 From: naylor.b.david at gmail.com (David Naylor) Date: Tue, 22 Nov 2011 21:39:46 +0200 Subject: [pypy-dev] Translating pypy-1.7 with -O0 Message-ID: <201111222139.49249.naylor.b.david@gmail.com> Hi, I'm trying to translate pypy-1.7 with optimisations at level 0. I get the following error: # /usr/local/bin/pypy translate.py --source --gcrootfinder=shadowstack -- thread -O0 targetpypystandalone.py [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 308, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/driver.py", line 809, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/driver.py", line 286, in _do [translation:ERROR] res = func() [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/driver.py", line 505, in task_database_c [translation:ERROR] database = cbuilder.build_database() [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/c/genc.py", line 143, in build_database [translation:ERROR] sandbox=self.config.translation.sandbox) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/translator/c/database.py", line 63, in __init__ [translation:ERROR] self.gctransformer = self.gcpolicy.transformerclass(translator) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/rpython/memory/gctransform/framework.py", line 151, in __init__ [translation:ERROR] GCClass, GC_PARAMS = choose_gc_from_config(translator.config) [translation:ERROR] File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- release-1.7/pypy/rpython/memory/gc/base.py", line 448, in choose_gc_from_config [translation:ERROR] config.translation.gc,)) [translation:ERROR] ValueError: unknown value for translation.gc: 'ref' Looking at base.py I see that ref and boehm GC are not in the list (and I thought -O0 used the boehm not the ref GC). I saw a previous message about this (re pypy-1.6) but no solution. Also: # file /usr/local/lib/libgc.so.1 /usr/local/lib/libgc.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, not stripped confirms I have the required library installed. Is there, perhaps, something I am missing? Regards -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: This is a digitally signed message part. URL: From frikker at gmail.com Tue Nov 22 20:42:53 2011 From: frikker at gmail.com (Blaine) Date: Tue, 22 Nov 2011 14:42:53 -0500 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: I've got a pure python and also equivalent C++ (with cython) cellular automata engine that I use as benchmarks. Thought the group would be interested in my results with the new pypy-1.7. The c++ engine is the original file and I built a thin interface with cython. I ported this to python and it is functionally equivalent. Most of the computation is simply bit operations, shifting, and writing to a wrap-around buffer. Interestingly, pypy chews through tens of millions of generations more easily than the C++ due to some latent memory issues that are probably a result of some error by me in coding. pypy doesn't seem to care, where the C++ engine brings my laptop to a crawl (choppy mouse, low responsiveness, etc) wth 10,000,000 generations. 1-dimensional cellular automata. Rule 30, 10,000 cells wide, 10,000 generations. This project is available on my github (print_ca.py is the benchmark i'm using). https://github.com/booherbg/PyAutomata *pypy-1.6 with pure python module* [blaine at macbook:~/src/PyAutomata Tue Nov 22] 30$ time pypy-1.6 print_ca.py 30 10000 10000 --quiet --python using pyautomata interface: PYTHON real 0m13.812s user 0m13.540s sys 0m0.170s *pypy-1.7 with pure python module (14% faster than pypy-1.6)* [blaine at macbook:~/src/PyAutomata Tue Nov 22] 29$ time pypy-1.7 print_ca.py 30 10000 10000 --quiet --python using pyautomata interface: PYTHON real 0m11.952s user 0m11.730s sys 0m0.140s *cpython with c++ interface - only ~2.5 times faster than pypy-1.7* [blaine at macbook:~/src/PyAutomata Tue Nov 22] 31$ time python print_ca.py 30 10000 10000 --quiet using pyautomata interface: C++ real 0m4.818s user 0m4.680s sys 0m0.060s *cpython with pure python interface - abysmally slow, ~160x slower than C++, ~64x slower than pypy-1.7* [blaine at macbook:~/src/PyAutomata Tue Nov 22] 32$ time python print_ca.py 30 10000 10000 --quiet --python using pyautomata interface: PYTHON real 12m50.927s user 12m48.380s sys 0m0.510s c++ build commands: [blaine at macbook:~/src/PyAutomata Tue Nov 22] 36$ make mkdir -p build/temp.linux-x86_64-2.6/ cython --cplus pyAutomata.pyx # The following was generated based on setup.py; ython setup.py build_ext --inplace gcc -march=native -O2 -pthread -fno-strict-aliasing -fwrapv -Wall -fPIC -I. -Iautomata-1.0.0/ -I/usr/include/python2.6 -c pyAutomata.cpp -o build/temp.linux-x86_64-2.6//pyAutomata.o gcc -march=native -O2 -pthread -fno-strict-aliasing -fwrapv -Wall -fPIC -I. -Iautomata-1.0.0/ -I/usr/include/python2.6 -c automata-1.0.0//Automata.cpp -o build/temp.linux-x86_64-2.6//automata-1.0.0//Automata.o g++ -march=native -lm -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6//pyAutomata.o build/temp.linux-x86_64-2.6//automata-1.0.0//Automata.o -lstdc++ -o pyAutomata.so Blaine On Tue, Nov 22, 2011 at 9:42 AM, Phyo Arkar wrote: > nice > > > On Tue, Nov 22, 2011 at 7:01 PM, Bruno Gola wrote: > >> Hi, >> >> Just to let you know I just uploaded the new release to the official PPA, >> so if you use Ubuntu you can install PyPy 1.7 using >> https://launchpad.net/~pypy/+archive/ppa >> >> Tests and feedback are welcome :-) >> >> Thanks, >> Bruno >> >> On Tue, Nov 22, 2011 at 7:36 AM, Phyo Arkar wrote: >> >>> I understand the complication of binary release. >>> >>> i did cross reference 1.0.0 and it works (starts) pypy fine,. But i am >>> not sure what will happen if something that uses crypting modules. >>> >>> may be release both versions of openssl (damn , python should have have >>> that openssl dependancy in standard modules :( ) >>> >>> >>> On Tue, Nov 22, 2011 at 3:40 PM, Armin Rigo wrote: >>> >>>> Hi Phyo, >>>> >>>> On Tue, Nov 22, 2011 at 09:38, Phyo Arkar >>>> wrote: >>>> > in my Gentoo based Sabayon they are already gone. and i believe next >>>> latest >>>> > Ubuntu may use 1.0.0 too. >>>> >>>> The point is of course that doing binary releases on Linux is always a >>>> bit of a problem, unless we ship everything as included static >>>> libraries, which is a bit nonsensical IMHO for a project like PyPy. >>>> >>>> Moreover, if openssl 1.0.0 is binary compatible with openssl 0.9.8, >>>> then I might rather put the blame on the openssl project for releasing >>>> two binary compatible versions with a different (un-cross-linkable) >>>> first number. But I will not do so: I guess that they are not >>>> supposed to be totally binary compatible, and there are subtle >>>> differences. >>>> >>>> Is there anyone that can (1) make sure the differences don't matter, >>>> possibly by looking at how the CPython source evolved; and (2) maybe >>>> come up with a way to say "link against openssl either 0.9.8 or 1.0.0" >>>> in the binary? >>>> >>>> >>>> A bient?t, >>>> >>>> Armin. >>>> >>> >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> >> > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From angelflow at yahoo.com Tue Nov 22 21:12:09 2011 From: angelflow at yahoo.com (Andy) Date: Tue, 22 Nov 2011 12:12:09 -0800 (PST) Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <20111121232313.GI27920@merlinux.eu> Message-ID: <1321992729.2464.YahooMailNeo@web111316.mail.gq1.yahoo.com> By "stackless features" do you mean just Stackless Python? Or does this limitation apply to any non-blocking system like gevent? ________________________________ From: Maciej Fijalkowski To: holger krekel Cc: PyPy Developer Mailing List Sent: Tuesday, November 22, 2011 3:47 AM Subject: Re: [pypy-dev] PyPy 1.7 - widening the sweet spot On Tue, Nov 22, 2011 at 1:23 AM, holger krekel wrote: > On Mon, Nov 21, 2011 at 12:35 +0200, Maciej Fijalkowski wrote: >> * PyPy now comes with stackless features enabled by default. However, >> ? any loop using stackless features will interrupt the JIT for now, so no real >> ? performance improvement for stackless-based programs. Contact pypy-dev for >> ? info how to help on removing this restriction. > > Yes, please, could you talk a bit more explicitely about what is involved > and what works/integrated and what doesn't? > > best, > holger > Everything works, but every instruction that can cause a switch (so pretty much using *any* of stackless features) will disable jitting for this loop. this is unideal, and I don't quite recall the exact reasons why, other than "more work is needed". Armin, can you elaborate? Cheers, fijal _______________________________________________ pypy-dev mailing list pypy-dev at python.org http://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 22 22:39:55 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 22 Nov 2011 23:39:55 +0200 Subject: [pypy-dev] Translating pypy-1.7 with -O0 In-Reply-To: <201111222139.49249.naylor.b.david@gmail.com> References: <201111222139.49249.naylor.b.david@gmail.com> Message-ID: On Tue, Nov 22, 2011 at 9:39 PM, David Naylor wrote: > Hi, > > I'm trying to translate pypy-1.7 with optimisations at level 0. ?I get the > following error: > > # /usr/local/bin/pypy translate.py --source --gcrootfinder=shadowstack -- > thread -O0 ?targetpypystandalone.py > > [translation:ERROR] Error: > [translation:ERROR] ?Traceback (most recent call last): > [translation:ERROR] ? ?File "translate.py", line 308, in main > [translation:ERROR] ? ? drv.proceed(goals) > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/translator/driver.py", line 809, in proceed > [translation:ERROR] ? ? return self._execute(goals, task_skip = > self._maybe_skip()) > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/translator/tool/taskengine.py", line 116, in _execute > [translation:ERROR] ? ? res = self._do(goal, taskcallable, *args, **kwds) > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/translator/driver.py", line 286, in _do > [translation:ERROR] ? ? res = func() > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/translator/driver.py", line 505, in task_database_c > [translation:ERROR] ? ? database = cbuilder.build_database() > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/translator/c/genc.py", line 143, in build_database > [translation:ERROR] ? ? sandbox=self.config.translation.sandbox) > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/translator/c/database.py", line 63, in __init__ > [translation:ERROR] ? ? self.gctransformer = > self.gcpolicy.transformerclass(translator) > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/rpython/memory/gctransform/framework.py", line 151, in > __init__ > [translation:ERROR] ? ? GCClass, GC_PARAMS = > choose_gc_from_config(translator.config) > [translation:ERROR] ? ?File "/tmp/home/DragonSA/ports/pypy/work/pypy-pypy- > release-1.7/pypy/rpython/memory/gc/base.py", line 448, in > choose_gc_from_config > [translation:ERROR] ? ? config.translation.gc,)) > [translation:ERROR] ?ValueError: unknown value for translation.gc: 'ref' > > Looking at base.py I see that ref and boehm GC are not in the list (and I > thought -O0 used the boehm not the ref GC). > > I saw a previous message about this (re pypy-1.6) but no solution. ?Also: > # file /usr/local/lib/libgc.so.1 > /usr/local/lib/libgc.so.1: ELF 64-bit LSB shared object, x86-64, version 1 > (FreeBSD), dynamically linked, not stripped > > confirms I have the required library installed. > > Is there, perhaps, something I am missing? > > Regards > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > Hey First, why are you doing that? -O0 is designed to only check if stuff compiles, not to be used. Second it seems weird it tries to use refcounting. Do you have boehm installed properly? Cheers, fijal From randall.leeds at gmail.com Wed Nov 23 09:06:41 2011 From: randall.leeds at gmail.com (Randall Leeds) Date: Wed, 23 Nov 2011 03:06:41 -0500 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: <1321992729.2464.YahooMailNeo@web111316.mail.gq1.yahoo.com> References: <20111121232313.GI27920@merlinux.eu> <1321992729.2464.YahooMailNeo@web111316.mail.gq1.yahoo.com> Message-ID: On Tue, Nov 22, 2011 at 15:12, Andy wrote: > By "stackless features" do you mean just Stackless Python? Or does this > limitation apply to any non-blocking system like gevent? I believe this means anything that using the continuation support. That includes anything importing greenlet, such as gevent and eventlet. (Aside, gevent is a cython extension so is currently unusable with pypy anyway.) > ________________________________ > From: Maciej Fijalkowski > To: holger krekel > Cc: PyPy Developer Mailing List > Sent: Tuesday, November 22, 2011 3:47 AM > Subject: Re: [pypy-dev] PyPy 1.7 - widening the sweet spot > > On Tue, Nov 22, 2011 at 1:23 AM, holger krekel wrote: >> On Mon, Nov 21, 2011 at 12:35 +0200, Maciej Fijalkowski wrote: >>> * PyPy now comes with stackless features enabled by default. However, >>> ? any loop using stackless features will interrupt the JIT for now, so no >>> real >>> ? performance improvement for stackless-based programs. Contact pypy-dev >>> for >>> ? info how to help on removing this restriction. >> >> Yes, please, could you talk a bit more explicitely about what is involved >> and what works/integrated and what doesn't? >> >> best, >> holger >> > > Everything works, but every instruction that can cause a switch (so > pretty much using *any* of stackless features) will disable jitting > for this loop. this is unideal, and I don't quite recall the exact > reasons why, other than "more work is needed". Armin, can you > elaborate? > > Cheers, > fijal > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > From cfbolz at gmx.de Wed Nov 23 10:10:46 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 23 Nov 2011 10:10:46 +0100 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: References: <1321892047.16283.13.camel@anglides.russel.org.uk> <1321893225.16283.24.camel@anglides.russel.org.uk> Message-ID: <4ECCB896.9020400@gmx.de> On 11/22/2011 08:42 PM, Blaine wrote: > I've got a pure python and also equivalent C++ (with cython) cellular > automata engine that I use as benchmarks. Thought the group would be > interested in my results with the new pypy-1.7. [snip] Very nice, thanks for sharing! Cheers, Carl Friedrich From arigo at tunes.org Wed Nov 23 11:40:23 2011 From: arigo at tunes.org (Armin Rigo) Date: Wed, 23 Nov 2011 11:40:23 +0100 Subject: [pypy-dev] PyPy 1.7 - widening the sweet spot In-Reply-To: <1321992729.2464.YahooMailNeo@web111316.mail.gq1.yahoo.com> References: <20111121232313.GI27920@merlinux.eu> <1321992729.2464.YahooMailNeo@web111316.mail.gq1.yahoo.com> Message-ID: Hi Andy, On Tue, Nov 22, 2011 at 21:12, Andy wrote: > By "stackless features" do you mean just Stackless Python? Or does this > limitation apply to any non-blocking system like gevent? By "stackless features" I mean anything based on the pypy-specific "_continuation" module, which include the "stackless" and the "greenlet" module. A bient?t, Armin. From arigo at tunes.org Wed Nov 23 12:23:43 2011 From: arigo at tunes.org (Armin Rigo) Date: Wed, 23 Nov 2011 12:23:43 +0100 Subject: [pypy-dev] Translating pypy-1.7 with -O0 In-Reply-To: References: <201111222139.49249.naylor.b.david@gmail.com> Message-ID: Hi David, You are specifying conflicting options: --gcrootfinder=shadowstack does not make sense with the Boehm GC. I fixed the source to report it properly as an error, rather than getting the strange crash later. It's unclear what you are trying to build and why, so I cannot give you the fixed command line that you mean to use... A bient?t, Armin. From tbaldridge at gmail.com Wed Nov 23 16:18:31 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Wed, 23 Nov 2011 09:18:31 -0600 Subject: [pypy-dev] Bytecode-less pypy languages Message-ID: So I've started work on my Clojure(lisp) in pypy interpreter again. This time I've decided to take a very materialistic approach to the design. As such my interpreter does not run off of bytecode, instead it directly interprets the Clojure structures, truely treating data as code. The idea is that every object in the system must have a evauate() method, and optionally a invoke(args) method. These methods are implemented as follows: most objects return themselves when evalute() is run def evaulates to itself, and when invoked, creates a named var that is bound to the 2nd form. E.g. (def foo 1) creates a var named foo that points to IntObj(1) user defined functions create local bindings, and then invoke their contents symbols search local bindings and global defs and then return the result through evaluate() lists evaluate all their contents, and then invoke the first item passing it the rest of the list as arguments With this simple approach, I've been able to implement many of the clojure concepts with just a few lines of code. It really works well. Here is my question. When I get a full program written, I'll have hundreds of objects each with evaluate() and invoke() all will be immutable, but there really isn't any sort of loop. We can implement loops via recur: (def (fn (x) (if (< x 100) (recur (inc x)) x))) But that's about the only loop in the interpreter. So I know in "normal" jits you would "greenlight" the instruction pointer of the byte code interpreter. Will PyPy not be happy with a lack of an instruction pointer? If I flag every single class as immutable, will it be able to still a decent jit? I guess, what I'm asking is when do green and red variables need to be defined, is it ever okay to not define them? Will the jit even work? Thank you for your time, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From santagada at gmail.com Wed Nov 23 16:39:13 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 23 Nov 2011 13:39:13 -0200 Subject: [pypy-dev] Bytecode-less pypy languages In-Reply-To: References: Message-ID: On Wed, Nov 23, 2011 at 1:18 PM, Timothy Baldridge wrote: > So I've started work on my Clojure(lisp) in pypy interpreter again. > This time I've decided to take a very materialistic approach to the > design. As such my interpreter does not run off of bytecode, instead > it directly interprets the Clojure structures, truely treating data as > code. The idea is that every object in the system must have a > evauate() method, and optionally a invoke(args) method. These methods > are implemented as follows: Nice. > most objects return themselves when evalute() is run > def evaulates to itself, and when invoked, creates a named var that is > bound to the 2nd form. E.g. (def foo 1) creates a var named foo that > points to IntObj(1) > user defined functions create local bindings, and then invoke their contents > symbols search local bindings and global defs and then return the > result through evaluate() > lists evaluate all their contents, and then invoke the first item > passing it the rest of the list as arguments > > With this simple approach, I've been able to implement many of the > clojure concepts with just a few lines of code. It really works well. > > Here is my question. When I get a full program written, I'll have > hundreds of objects each with evaluate() and invoke() all will be > immutable, but there really isn't any sort of loop. We can implement > loops via recur: > > (def (fn (x) > ? ? ? ? ? (if (< x 100) > ? ? ? ? ? ? ?(recur (inc x)) > ? ? ? ? ? ? ?x))) > > But that's about the only loop in the interpreter. > > So I know in "normal" jits you would "greenlight" the instruction > pointer of the byte code interpreter. Will PyPy not be happy with a > lack of an instruction pointer? If I flag every single class as > immutable, will it be able to still a decent jit? I guess, what I'm > asking is when do green and red variables need to be defined, is it > ever okay to not define them? Will the jit even work? IIRC armin wanted to try the jit with the javascript interpreter, that a long time ago interpreted the AST directly. But we never end up doing this, but I think it is a great idea for clojure and specially for simple languages were you don't want to waste time implementing a bytecode vm. -- Leonardo Santagada From cfbolz at gmx.de Wed Nov 23 16:46:29 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 23 Nov 2011 16:46:29 +0100 Subject: [pypy-dev] Bytecode-less pypy languages In-Reply-To: References: Message-ID: <4ECD1555.9040806@gmx.de> On 11/23/2011 04:18 PM, Timothy Baldridge wrote: > So I've started work on my Clojure(lisp) in pypy interpreter again. > This time I've decided to take a very materialistic approach to the > design. As such my interpreter does not run off of bytecode, instead > it directly interprets the Clojure structures, truely treating data as > code. The idea is that every object in the system must have a > evauate() method, and optionally a invoke(args) method. These methods > are implemented as follows: > > most objects return themselves when evalute() is run > def evaulates to itself, and when invoked, creates a named var that is > bound to the 2nd form. E.g. (def foo 1) creates a var named foo that > points to IntObj(1) > user defined functions create local bindings, and then invoke their contents > symbols search local bindings and global defs and then return the > result through evaluate() > lists evaluate all their contents, and then invoke the first item > passing it the rest of the list as arguments > > With this simple approach, I've been able to implement many of the > clojure concepts with just a few lines of code. It really works well. > > Here is my question. When I get a full program written, I'll have > hundreds of objects each with evaluate() and invoke() all will be > immutable, but there really isn't any sort of loop. We can implement > loops via recur: > > (def (fn (x) > (if (< x 100) > (recur (inc x)) > x))) > > But that's about the only loop in the interpreter. > > So I know in "normal" jits you would "greenlight" the instruction > pointer of the byte code interpreter. Will PyPy not be happy with a > lack of an instruction pointer? If I flag every single class as > immutable, will it be able to still a decent jit? I guess, what I'm > asking is when do green and red variables need to be defined, is it > ever okay to not define them? Will the jit even work? There is no deep reason why the JIT won't work. You might have to have one or two small hacks in the main interpreter loop for the JIT, but should not be too annoying. In fact, the Prolog interpreter in RPython works quite similarly. Cheers, Carl Friedrich From naylor.b.david at gmail.com Wed Nov 23 19:53:05 2011 From: naylor.b.david at gmail.com (David Naylor) Date: Wed, 23 Nov 2011 20:53:05 +0200 Subject: [pypy-dev] Translating pypy-1.7 with -O0 In-Reply-To: References: <201111222139.49249.naylor.b.david@gmail.com> Message-ID: <201111232053.09826.naylor.b.david@gmail.com> On Wednesday, 23 November 2011 13:23:43 you wrote: > Hi David, Hi > You are specifying conflicting options: --gcrootfinder=shadowstack > does not make sense with the Boehm GC. Removing --gcrootfinder=shadowstack fixes the problem for me. Thanks. > I fixed the source to report > it properly as an error, rather than getting the strange crash later. > It's unclear what you are trying to build and why, so I cannot give > you the fixed command line that you mean to use... I am building a port of pypy for FreeBSD so I am busy testing some purmutations, such as various optimisation levels. Boehm has an extra dependency so needs specific testing and since it was easy to change the optimisation level I opted to test translate a -O0 pypy. I needed to patch an assumption about the dl library (which FreeBSD doesn't have) and the translation works. The binary, however, does not work and core dumps when started with: #0 0x00000008010debf8 in GC_FreeBSDGetDataStart () from /usr/local/lib/libgc.so.1 #1 0x00000008010dec69 in GC_register_data_segments () from /usr/local/lib/libgc.so.1 #2 0x00000008010dd815 in GC_init_inner () from /usr/local/lib/libgc.so.1 #3 0x0000000000412949 in pypy_main_function () #4 0x000000000041178e in _start () I'm happy to accept this as a bug in libgc (and ignore it). Regards -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-pypy::rpython::tool::rffi_platform.py Type: text/x-python Size: 464 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: This is a digitally signed message part. URL: From arigo at tunes.org Thu Nov 24 13:10:11 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 24 Nov 2011 13:10:11 +0100 Subject: [pypy-dev] Translating pypy-1.7 with -O0 In-Reply-To: <201111232053.09826.naylor.b.david@gmail.com> References: <201111222139.49249.naylor.b.david@gmail.com> <201111232053.09826.naylor.b.david@gmail.com> Message-ID: Hi David, On Wed, Nov 23, 2011 at 19:53, David Naylor wrote: > The binary, however, does not work and core dumps when started with: This looks like a misuse of libgc. If you really care, you need to read the docs of Boehm to see if they say anything specific to FreeBSD and to the way the generated code calls it, e.g. maybe we are missing some initialization function that is required on FreeBSD but not on Linux. (But again, I don't think we care about Boehm that much, except for testing.) A bient?t, Armin. From jfcgauss at gmail.com Fri Nov 25 17:06:13 2011 From: jfcgauss at gmail.com (Serhat Sevki Dincer) Date: Fri, 25 Nov 2011 18:06:13 +0200 Subject: [pypy-dev] Real world comparison with native application & cpython Message-ID: I wrote a tiny grep with multi-line match support, and compared its speed under pypy 1.7 with grep and CPython 2.7.1 (on ubuntu 11.04 laptop). No special algorithm/implementation is employed; it is bare re module. input: Plone 4.1.2 eggs directory, size 286mb, possible processed input size is about 75mb, processed 3958 files total commands: time mgrp -lcrN '\.py$' for . takes 1.95s time python2.7 /usr/local/bin/mgrp -lcrN '\.py$' for . takes 1.45s time grep -lcr --color=none --include='*.py' for . takes 0.6s Is the input too small to see the benefits of pypy? -------------- next part -------------- A non-text attachment was scrubbed... Name: mgrp Type: application/octet-stream Size: 6566 bytes Desc: not available URL: From benjamin at python.org Fri Nov 25 17:15:19 2011 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 25 Nov 2011 10:15:19 -0600 Subject: [pypy-dev] Real world comparison with native application & cpython In-Reply-To: References: Message-ID: 2011/11/25 Serhat Sevki Dincer : > I wrote a tiny grep with multi-line match support, and compared its > speed under pypy 1.7 with grep and CPython 2.7.1 (on ubuntu 11.04 > laptop). > No special algorithm/implementation is employed; it is bare re module. > > input: Plone 4.1.2 eggs directory, size 286mb, possible processed > input size is about 75mb, processed 3958 files total > > commands: > > time mgrp -lcrN '\.py$' for . > takes 1.95s > > time python2.7 /usr/local/bin/mgrp -lcrN '\.py$' for . > takes 1.45s > > time grep -lcr --color=none --include='*.py' for . > takes 0.6s > > Is the input too small to see the benefits of pypy? It would instructive to see the code, but if what you're expecting it to be as fast as grep, think again. It has extremely well-tuned clever algorithms. -- Regards, Benjamin From piotr.skamruk at gmail.com Fri Nov 25 17:22:08 2011 From: piotr.skamruk at gmail.com (Piotr Skamruk) Date: Fri, 25 Nov 2011 17:22:08 +0100 Subject: [pypy-dev] Real world comparison with native application & cpython In-Reply-To: References: Message-ID: Serhat had probably in mind that pypy1.7 is slower than cpython2.7 in this test. Serhat: add this as bug to bugs.pypy.org 2011/11/25 Benjamin Peterson : > 2011/11/25 Serhat Sevki Dincer : >> I wrote a tiny grep with multi-line match support, and compared its >> speed under pypy 1.7 with grep and CPython 2.7.1 (on ubuntu 11.04 >> laptop). >> No special algorithm/implementation is employed; it is bare re module. >> >> input: Plone 4.1.2 eggs directory, size 286mb, possible processed >> input size is about 75mb, processed 3958 files total >> >> commands: >> >> time mgrp -lcrN '\.py$' for . >> takes 1.95s >> >> time python2.7 /usr/local/bin/mgrp -lcrN '\.py$' for . >> takes 1.45s >> >> time grep -lcr --color=none --include='*.py' for . >> takes 0.6s >> >> Is the input too small to see the benefits of pypy? > > It would instructive to see the code, but if what you're expecting it > to be as fast as grep, think again. It has extremely well-tuned clever > algorithms. > > > -- > Regards, > Benjamin > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From jfcgauss at gmail.com Fri Nov 25 17:31:38 2011 From: jfcgauss at gmail.com (Serhat Sevki Dincer) Date: Fri, 25 Nov 2011 18:31:38 +0200 Subject: [pypy-dev] Real world comparison with native application & cpython In-Reply-To: References: Message-ID: https://bugs.pypy.org/issue940 On Fri, Nov 25, 2011 at 6:22 PM, Piotr Skamruk wrote: > Serhat had probably in mind that pypy1.7 is slower than cpython2.7 in this test. > > Serhat: add this as bug to bugs.pypy.org From yselivanov.ml at gmail.com Fri Nov 25 17:39:16 2011 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 25 Nov 2011 11:39:16 -0500 Subject: [pypy-dev] Real world comparison with native application & cpython In-Reply-To: References: Message-ID: <4B38F75E-ED7E-4523-AE08-57B85B89A6D8@gmail.com> Maybe it's a good idea to find the bottleneck in the test and extend speed.pypy suite? The more performance tests it hosts the better. -Yury On 2011-11-25, at 11:31 AM, Serhat Sevki Dincer wrote: > https://bugs.pypy.org/issue940 > > On Fri, Nov 25, 2011 at 6:22 PM, Piotr Skamruk wrote: >> Serhat had probably in mind that pypy1.7 is slower than cpython2.7 in this test. >> >> Serhat: add this as bug to bugs.pypy.org > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Sun Nov 27 23:59:13 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 27 Nov 2011 23:59:13 +0100 Subject: [pypy-dev] Win32 build of the release 1.7 Message-ID: Hi all, We have a tentative Win32 build of PyPy 1.7 including both the JIT and Stackless features (the '_continuation' primitive module and the app-level 'stackless' and 'greenlet' modules). It built and passed a good proportion of tests. If you are interested in helping, could a few people try to download it and tell us if it works, and if something is obviously missing? Thank you! http://buildbot.pypy.org/nightly/release-1.7.x/pypy-c-jit-49856-930f0bc4125a-win32.zip A bient?t, Armin. From 1989lzhh at gmail.com Mon Nov 28 12:36:06 2011 From: 1989lzhh at gmail.com (=?GB2312?B?wfXV8bqj?=) Date: Mon, 28 Nov 2011 19:36:06 +0800 Subject: [pypy-dev] Win32 build of the release 1.7 In-Reply-To: References: Message-ID: Hi, Thanks for you guys to create such amazing interpreter. I try the numpypy module and find this. import numpypy as np a=np.array([1,2,3]) a**2 # here the result is [0, 0 ,0], I think this is a bug a*2 # here the result is [2,4,6] Regards, Liu Zhenhai 2011/11/28 Armin Rigo > Hi all, > > We have a tentative Win32 build of PyPy 1.7 including both the JIT and > Stackless features (the '_continuation' primitive module and the > app-level 'stackless' and 'greenlet' modules). It built and passed a > good proportion of tests. > > If you are interested in helping, could a few people try to download > it and tell us if it works, and if something is obviously missing? > Thank you! > > > http://buildbot.pypy.org/nightly/release-1.7.x/pypy-c-jit-49856-930f0bc4125a-win32.zip > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Mon Nov 28 12:40:58 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 28 Nov 2011 06:40:58 -0500 Subject: [pypy-dev] Win32 build of the release 1.7 In-Reply-To: References: Message-ID: On Mon, Nov 28, 2011 at 6:36 AM, ??? <1989lzhh at gmail.com> wrote: > Hi, > > Thanks for you guys to create such amazing interpreter. > > I try the numpypy module and find this. > > import numpypy as np > a=np.array([1,2,3]) > a**2 # here the result is [0, 0 ,0], I think this is a bug > a*2 # here the result is [2,4,6] > > > Regards, > Liu Zhenhai > > > 2011/11/28 Armin Rigo > >> Hi all, >> >> We have a tentative Win32 build of PyPy 1.7 including both the JIT and >> Stackless features (the '_continuation' primitive module and the >> app-level 'stackless' and 'greenlet' modules). It built and passed a >> good proportion of tests. >> >> If you are interested in helping, could a few people try to download >> it and tell us if it works, and if something is obviously missing? >> Thank you! >> >> >> http://buildbot.pypy.org/nightly/release-1.7.x/pypy-c-jit-49856-930f0bc4125a-win32.zip >> >> >> A bient?t, >> >> Armin. >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > This appears to be totally broken independently of windows, on Linux I'm getting something that looks like sys.maxint instead of 0. Can you file a bug in our tracker for this? It's not going to be a release blocker for windows, but it'll be fixed in the next release. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide.del.vento at gmail.com Mon Nov 28 20:14:58 2011 From: davide.del.vento at gmail.com (Davide Del Vento) Date: Mon, 28 Nov 2011 12:14:58 -0700 Subject: [pypy-dev] PyPy related internship at NCAR Message-ID: Folks, NCAR has an internship program in computer science (see http://www2.cisl.ucar.edu/siparcs for details - note that interns are well paid and have many benefits). I'm mentor for a PyPy internship: http://www2.cisl.ucar.edu/siparcs/opportunities/ad The internship is open to university students enrolled an USA university. Feel free to contact me if you have any questions, or simply DO APPLY :-) Best, Davide From coolbutuseless at gmail.com Mon Nov 28 21:37:33 2011 From: coolbutuseless at gmail.com (mike c) Date: Tue, 29 Nov 2011 06:37:33 +1000 Subject: [pypy-dev] Win32 build of the release 1.7 In-Reply-To: References: Message-ID: On Mon, Nov 28, 2011 at 9:40 PM, Alex Gaynor wrote: > > This appears to be totally broken independently of windows, on Linux I'm > getting something that looks like sys.maxint instead of 0. Can you file a > bug in our tracker for this? It's not going to be a release blocker for > windows, but it'll be fixed in the next release. > > On OSX Snow Leopard pypy1.7 >>>> import numpypy as numpy >>>> a = numpy.array([1,2,3]) >>>> a**2 array([4607182418800017408, 4607182418800017408, 4607182418800017408], dtype=int64) mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Tue Nov 29 02:19:30 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 28 Nov 2011 20:19:30 -0500 Subject: [pypy-dev] Sandbox startup time Message-ID: <4ED43322.3020208@nedbatchelder.com> Hi all, I'm trying to use the PyPy sandbox to execute student code on a server. I've got it all working, but it takes a few seconds to start the sandbox process, so the response time from the server is poor. There are a few issues: 1) The sandbox makes a lot of system calls as part of the Python startup. I understand how to reduce these somewhat by creating dummy resources. 2) The sandbox invokes the C compiler to determine system characteristics. I got a patch from Geoff Thomas as MIT (they used the sandbox for labs in a security course). He found that this change got rid of his C compiler invocation: --- clean/pypy-pypy-release-1.6/pypy/rlib/rmarshal.py 2011-08-15 11:10:35.000000000 -0400 +++ pypy-pypy-release-1.6/pypy/rlib/rmarshal.py 2011-09-27 19:32:51.339470297 -0400 @@ -195,7 +195,7 @@ def dump_float(buf, x): buf.append(TYPE_FLOAT) - s = formatd(x, 'g', 17) + s = '%f' % (x) buf.append(chr(len(s))) buf += s add_dumper(annmodel.SomeFloat(), dump_float) I tried this change, and it reduced the number of compiler calls from three to one. Before the change: [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused /tmp/usession-default-4/platcheck_7.c -o /tmp/usession-default-4/platcheck_7.o Warning: cannot find your CPU L2 cache size in /proc/cpuinfo [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused -I/home/ned/pypy/pypy/translator/c /tmp/usession-default-4/module_cache/module_0.c -o /tmp/usession-default-4/module_cache/module_0.o [platform:execute] gcc -shared /tmp/usession-default-4/module_cache/module_0.o -pthread -lrt -Wl,--export-dynamic,--version-script=/tmp/usession-default-4/dynamic-symbols-0 -o /tmp/usession-default-4/shared_cache/externmod.so After the change: [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused /tmp/usession-default-3/platcheck_7.c -o /tmp/usession-default-3/platcheck_7.o Warning: cannot find your CPU L2 cache size in /proc/cpuinfo I think this compile is to determine the size of floats? I don't understand why the sandbox needs to determine this dynamically every time it's run, and I don't understand how to get it not to, and I don't know what bad effects the patch has. I also don't know what the remaining compile does or how to disable it. If someone could mentor me through this process, I would appreciate it. Email or IRC are fine channels for me. --Ned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris0wj at gmail.com Tue Nov 29 14:58:20 2011 From: chris0wj at gmail.com (Chris Wj) Date: Tue, 29 Nov 2011 08:58:20 -0500 Subject: [pypy-dev] Win32 build of the release 1.7 In-Reply-To: References: Message-ID: Similar experience with **2 for me on linux x64. On Mon, Nov 28, 2011 at 3:37 PM, mike c wrote: > > > On Mon, Nov 28, 2011 at 9:40 PM, Alex Gaynor wrote: > >> >> This appears to be totally broken independently of windows, on Linux I'm >> getting something that looks like sys.maxint instead of 0. Can you file a >> bug in our tracker for this? It's not going to be a release blocker for >> windows, but it'll be fixed in the next release. >> >> > On OSX Snow Leopard pypy1.7 > > >>>> import numpypy as numpy > >>>> a = numpy.array([1,2,3]) > >>>> a**2 > array([4607182418800017408, 4607182418800017408, 4607182418800017408], > dtype=int64) > > mike > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 29 15:02:22 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 29 Nov 2011 16:02:22 +0200 Subject: [pypy-dev] Win32 build of the release 1.7 In-Reply-To: References: Message-ID: On Tue, Nov 29, 2011 at 3:58 PM, Chris Wj wrote: > Similar experience with **2 for me on linux x64. This should be fixed by now. You can check tomorrow nightly > > On Mon, Nov 28, 2011 at 3:37 PM, mike c wrote: >> >> >> >> On Mon, Nov 28, 2011 at 9:40 PM, Alex Gaynor >> wrote: >>> >>> >>> This appears to be totally broken independently of windows, on Linux I'm >>> getting something that looks like sys.maxint instead of 0. ?Can you file a >>> bug in our tracker for this? ?It's not going to be a release blocker for >>> windows, but it'll be fixed in the next release. >>> >> >> On OSX Snow Leopard pypy1.7 >> >> >>>> import numpypy as numpy >> >>>> a = numpy.array([1,2,3]) >> >>>> a**2 >> array([4607182418800017408, 4607182418800017408, 4607182418800017408], >> dtype=int64) >> >> mike >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From arigo at tunes.org Tue Nov 29 15:43:24 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 29 Nov 2011 15:43:24 +0100 Subject: [pypy-dev] Win32 build of the release 1.7 In-Reply-To: References: Message-ID: Hi, The build has been "officialized" and is now available from https://bitbucket.org/pypy/pypy/downloads . A bient?t, Armin. From tbaldridge at gmail.com Tue Nov 29 21:04:42 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 29 Nov 2011 14:04:42 -0600 Subject: [pypy-dev] Optimizing Append Only structures Message-ID: I have a dictionary (in RPython) that looks something like this: class MyDict(Obj): def add(self, k, v): newdict = self.dict.copy() newdict[k] = v self.dict = newdict def get(self, k): d = self.dict return MyDict._static_get(d, k) @staticmethod @purefunction def _static_get(d, k): if d not in k: return None return d[k] I'm trying to figure out the best way to optimize this. As you see, this structure is "append only". That is if mydict.get("foo") returns a value it will always return that value, for all time. In that sense, the function is pure. If however, the function returns None, then it could change in the future. This is for my Clojure on PyPy implementation. The idea is that types can be extended via protocols, at runtime, at any point in the program's execution. However, once a given type is extended to support a given interface (or protocol) it will never be able to be changed. That is, once we extend Foo so that it implements Foo.count() it will implement Foo.count() for all time. Any thoughts on how to optimize this further for the jit? I'd like to promote the value of get() to a constant, but I can only do that as long as it is not None. After Foo is extended, I'd like the jit to re-generate the jitted code to remove all guards, hopefully saving a few cycles. Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From alex.gaynor at gmail.com Tue Nov 29 21:12:18 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 29 Nov 2011 15:12:18 -0500 Subject: [pypy-dev] Optimizing Append Only structures In-Reply-To: References: Message-ID: On Tue, Nov 29, 2011 at 3:04 PM, Timothy Baldridge wrote: > I have a dictionary (in RPython) that looks something like this: > > > class MyDict(Obj): > def add(self, k, v): > newdict = self.dict.copy() > newdict[k] = v > self.dict = newdict > def get(self, k): > d = self.dict > return MyDict._static_get(d, k) > @staticmethod > @purefunction > def _static_get(d, k): > if d not in k: > return None > return d[k] > > > I'm trying to figure out the best way to optimize this. As you see, > this structure is "append only". That is if mydict.get("foo") returns > a value it will always return that value, for all time. In that sense, > the function is pure. If however, the function returns None, then it > could change in the future. > > This is for my Clojure on PyPy implementation. The idea is that types > can be extended via protocols, at runtime, at any point in the > program's execution. However, once a given type is extended to support > a given interface (or protocol) it will never be able to be changed. > That is, once we extend Foo so that it implements Foo.count() it will > implement Foo.count() for all time. > > Any thoughts on how to optimize this further for the jit? I'd like to > promote the value of get() to a constant, but I can only do that as > long as it is not None. After Foo is extended, I'd like the jit to > re-generate the jitted code to remove all guards, hopefully saving a > few cycles. > > Timothy > > > -- > ?One of the main causes of the fall of the Roman Empire was > that?lacking zero?they had no way to indicate successful termination > of their C programs.? > (Robert Firth) > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Sounds like the idea you're looking for is basically out-of-line guards. Basically the idea of these is you have field which rarely changes, and when it does you should regenerate all code. So you would have an object with the dict and a signature, and you could mutate the dict, but whenever you did, you'd need to update the signature. PyPy's module-dictionary implementation shows an example of this: https://bitbucket.org/pypy/pypy/src/default/pypy/objspace/std/celldict.py Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 29 21:15:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 29 Nov 2011 22:15:49 +0200 Subject: [pypy-dev] Optimizing Append Only structures In-Reply-To: References: Message-ID: On Tue, Nov 29, 2011 at 10:12 PM, Alex Gaynor wrote: > > > On Tue, Nov 29, 2011 at 3:04 PM, Timothy Baldridge > wrote: >> >> I have a dictionary (in RPython) that looks something like this: >> >> >> class MyDict(Obj): >> ? ?def add(self, k, v): >> ? ? ? ? newdict = self.dict.copy() >> ? ? ? ? newdict[k] = v >> ? ? ? ? self.dict = newdict >> ? ?def get(self, k): >> ? ? ? ?d = self.dict >> ? ? ? ?return MyDict._static_get(d, k) >> ? @staticmethod >> ? @purefunction >> ? def _static_get(d, k): >> ? ? ? ?if d not in k: >> ? ? ? ? ? ?return None >> ? ? ? ?return d[k] >> >> >> I'm trying to figure out the best way to optimize this. As you see, >> this structure is "append only". That is if mydict.get("foo") returns >> a value it will always return that value, for all time. In that sense, >> the function is pure. If however, the function returns None, then it >> could change in the future. >> >> This is for my Clojure on PyPy implementation. The idea is that types >> can be extended via protocols, at runtime, at any point in the >> program's execution. However, once a given type is extended to support >> a given interface (or protocol) it will never be able to be changed. >> That is, once we extend Foo so that it implements Foo.count() it will >> implement Foo.count() for all time. >> >> Any thoughts on how to optimize this further for the jit? I'd like to >> promote the value of get() to a constant, but I can only do that as >> long as it is not None. After Foo is extended, I'd like the jit to >> re-generate the jitted code to remove all guards, hopefully saving a >> few cycles. >> >> Timothy >> >> >> -- >> ?One of the main causes of the fall of the Roman Empire was >> that?lacking zero?they had no way to indicate successful termination >> of their C programs.? >> (Robert Firth) >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > > Sounds like the idea you're looking for is basically out-of-line guards. > ?Basically the idea of these is you have ?field which rarely changes, and > when it does you should regenerate all code. ?So you would have an object > with the dict and a signature, and you could mutate the dict, but whenever > you did, you'd need to update the signature. ?PyPy's module-dictionary > implementation shows an example of > this:?https://bitbucket.org/pypy/pypy/src/default/pypy/objspace/std/celldict.py > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Why do you always make a copy of a dict btw? From romain.py at gmail.com Tue Nov 29 21:28:44 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Tue, 29 Nov 2011 21:28:44 +0100 Subject: [pypy-dev] Optimizing Append Only structures In-Reply-To: References: Message-ID: Probably because he (as a clojure developer) likes immutability of data structures. On Tue, Nov 29, 2011 at 9:15 PM, Maciej Fijalkowski wrote: > On Tue, Nov 29, 2011 at 10:12 PM, Alex Gaynor > wrote: > > > > > > On Tue, Nov 29, 2011 at 3:04 PM, Timothy Baldridge > > > wrote: > >> > >> I have a dictionary (in RPython) that looks something like this: > >> > >> > >> class MyDict(Obj): > >> def add(self, k, v): > >> newdict = self.dict.copy() > >> newdict[k] = v > >> self.dict = newdict > >> def get(self, k): > >> d = self.dict > >> return MyDict._static_get(d, k) > >> @staticmethod > >> @purefunction > >> def _static_get(d, k): > >> if d not in k: > >> return None > >> return d[k] > >> > >> > >> I'm trying to figure out the best way to optimize this. As you see, > >> this structure is "append only". That is if mydict.get("foo") returns > >> a value it will always return that value, for all time. In that sense, > >> the function is pure. If however, the function returns None, then it > >> could change in the future. > >> > >> This is for my Clojure on PyPy implementation. The idea is that types > >> can be extended via protocols, at runtime, at any point in the > >> program's execution. However, once a given type is extended to support > >> a given interface (or protocol) it will never be able to be changed. > >> That is, once we extend Foo so that it implements Foo.count() it will > >> implement Foo.count() for all time. > >> > >> Any thoughts on how to optimize this further for the jit? I'd like to > >> promote the value of get() to a constant, but I can only do that as > >> long as it is not None. After Foo is extended, I'd like the jit to > >> re-generate the jitted code to remove all guards, hopefully saving a > >> few cycles. > >> > >> Timothy > >> > >> > >> -- > >> ?One of the main causes of the fall of the Roman Empire was > >> that?lacking zero?they had no way to indicate successful termination > >> of their C programs.? > >> (Robert Firth) > >> _______________________________________________ > >> pypy-dev mailing list > >> pypy-dev at python.org > >> http://mail.python.org/mailman/listinfo/pypy-dev > > > > > > Sounds like the idea you're looking for is basically out-of-line guards. > > Basically the idea of these is you have field which rarely changes, and > > when it does you should regenerate all code. So you would have an object > > with the dict and a signature, and you could mutate the dict, but > whenever > > you did, you'd need to update the signature. PyPy's module-dictionary > > implementation shows an example of > > this: > https://bitbucket.org/pypy/pypy/src/default/pypy/objspace/std/celldict.py > > > > Alex > > > > -- > > "I disapprove of what you say, but I will defend to the death your right > to > > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > > "The people's good is the highest law." -- Cicero > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > > > > Why do you always make a copy of a dict btw? > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yeomanyaacov at gmail.com Tue Nov 29 23:20:03 2011 From: yeomanyaacov at gmail.com (Yaacov Finkelman) Date: Tue, 29 Nov 2011 17:20:03 -0500 Subject: [pypy-dev] Pip, Pypy, and Win7 Message-ID: I've been a complete fanboy to Pypy for the past couple of months. After the official Sprint started I was checking the blog every couple of hours to see what wonderful and fascinating ideas people were working on implementing. I just want to thank you all for helping with this fascinating project. I was hoping to get some help. I have been trying to follow these (http://doc.pypy.org/en/latest/getting-started.html#installing-pypy) instructions to install pip on Pypy on my Win7 computer. $ curl -O http://python-distribute.org/distribute_setup.py So I downloaded that file and saved it to my Pypy directory "C:\pypy-1.7\" $ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py So I downloaded that file and saved it to my Pypy directory "C:\pypy-1.7\" $ ./pypy-1.6/bin/pypy distribute_setup.py Run this file with Pypy. (Runs with no errors) $ ./pypy-1.6/bin/pypy get-pip.py Run that file with Pypy. (Runs with no errors) $ ./pypy-1.6/bin/pip install pygments # for example I am not sure how to translate this Instruction. There is no executable Pip in my Pypy folder. How do I actually install something with pip? Jacob From santagada at gmail.com Tue Nov 29 23:29:56 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 29 Nov 2011 20:29:56 -0200 Subject: [pypy-dev] Pip, Pypy, and Win7 In-Reply-To: References: Message-ID: On Tue, Nov 29, 2011 at 8:20 PM, Yaacov Finkelman wrote: > $ ./pypy-1.6/bin/pip install pygments # for example > > I am not sure how to translate this Instruction. There is no > executable Pip in my Pypy folder. How do I actually install something > with pip? > > Jacob > first, try pypy-1.7 that was just released for windows. Also in windows I think pip will be in a subfolder named Scripts no bin. -- Leonardo Santagada From yeomanyaacov at gmail.com Tue Nov 29 23:53:07 2011 From: yeomanyaacov at gmail.com (Yaacov Finkelman) Date: Tue, 29 Nov 2011 17:53:07 -0500 Subject: [pypy-dev] Pip, Pypy, and Win7 In-Reply-To: References: Message-ID: Thank You! pip.exe and pip-2.7.exe are indeed in a subfolder called bin. Can we add that to the new version of the docs? When should I be using pip vs. using pip-2.7? Thank you again. Jacob On Tue, Nov 29, 2011 at 5:29 PM, Leonardo Santagada wrote: > On Tue, Nov 29, 2011 at 8:20 PM, Yaacov Finkelman > wrote: >> $ ./pypy-1.6/bin/pip install pygments # for example >> >> I am not sure how to translate this Instruction. There is no >> executable Pip in my Pypy folder. How do I actually install something >> with pip? >> >> Jacob >> > > first, try pypy-1.7 that was just released for windows. Also in > windows I think pip will be in a subfolder named Scripts no bin. > > > -- > Leonardo Santagada > From yeomanyaacov at gmail.com Wed Nov 30 00:35:47 2011 From: yeomanyaacov at gmail.com (Yaacov Finkelman) Date: Tue, 29 Nov 2011 18:35:47 -0500 Subject: [pypy-dev] PIL, Pypy-1.7, and Win7 Message-ID: Now the pip is working I was trying to use it to install PIL part of long error message is below. I am using Pypy-1.7 Any help? Jacob File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\build_ext.py", line 314, in run force=self.force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\ccompiler.py", line 10 54, in new_compiler return klass(None, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", l ine 309, in __init__ CygwinCCompiler.__init__ (self, verbose, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", l ine 99, in __init__ (status, details) = check_config_h() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", l ine 383, in check_config_h fn = sysconfig.get_config_h_filename() AttributeError: 'module' object has no attribute 'get_config_h_filename' Complete output from command C:\pypy-1.7\pypy.exe -c "import setuptools;__fi le__='c:\\pypy-1.7\\bin\\build\\PIL\\setup.py';exec(compile(open(__file__).read( ).replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally -managed --record c:\users\jlap\appdata\local\temp\pip-zpbnsd-record\install-rec ord.txt: WARNING: '' not a valid package name; please use only.-separated package nam es in setup.py running install running build running build_py creating build creating build\lib.win32-2.7 copying PIL\ArgImagePlugin.py -> build\lib.win32-2.7 copying PIL\BdfFontFile.py -> build\lib.win32-2.7 copying PIL\BmpImagePlugin.py -> build\lib.win32-2.7 copying PIL\BufrStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\ContainerIO.py -> build\lib.win32-2.7 copying PIL\CurImagePlugin.py -> build\lib.win32-2.7 copying PIL\DcxImagePlugin.py -> build\lib.win32-2.7 copying PIL\EpsImagePlugin.py -> build\lib.win32-2.7 copying PIL\ExifTags.py -> build\lib.win32-2.7 copying PIL\FitsStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\FliImagePlugin.py -> build\lib.win32-2.7 copying PIL\FontFile.py -> build\lib.win32-2.7 copying PIL\FpxImagePlugin.py -> build\lib.win32-2.7 copying PIL\GbrImagePlugin.py -> build\lib.win32-2.7 copying PIL\GdImageFile.py -> build\lib.win32-2.7 copying PIL\GifImagePlugin.py -> build\lib.win32-2.7 copying PIL\GimpGradientFile.py -> build\lib.win32-2.7 copying PIL\GimpPaletteFile.py -> build\lib.win32-2.7 copying PIL\GribStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\Hdf5StubImagePlugin.py -> build\lib.win32-2.7 copying PIL\IcnsImagePlugin.py -> build\lib.win32-2.7 copying PIL\IcoImagePlugin.py -> build\lib.win32-2.7 copying PIL\Image.py -> build\lib.win32-2.7 copying PIL\ImageChops.py -> build\lib.win32-2.7 copying PIL\ImageCms.py -> build\lib.win32-2.7 copying PIL\ImageColor.py -> build\lib.win32-2.7 copying PIL\ImageDraw.py -> build\lib.win32-2.7 copying PIL\ImageDraw2.py -> build\lib.win32-2.7 copying PIL\ImageEnhance.py -> build\lib.win32-2.7 copying PIL\ImageFile.py -> build\lib.win32-2.7 copying PIL\ImageFileIO.py -> build\lib.win32-2.7 copying PIL\ImageFilter.py -> build\lib.win32-2.7 copying PIL\ImageFont.py -> build\lib.win32-2.7 copying PIL\ImageGL.py -> build\lib.win32-2.7 copying PIL\ImageGrab.py -> build\lib.win32-2.7 copying PIL\ImageMath.py -> build\lib.win32-2.7 copying PIL\ImageMode.py -> build\lib.win32-2.7 copying PIL\ImageOps.py -> build\lib.win32-2.7 copying PIL\ImagePalette.py -> build\lib.win32-2.7 copying PIL\ImagePath.py -> build\lib.win32-2.7 copying PIL\ImageQt.py -> build\lib.win32-2.7 copying PIL\ImageSequence.py -> build\lib.win32-2.7 copying PIL\ImageShow.py -> build\lib.win32-2.7 copying PIL\ImageStat.py -> build\lib.win32-2.7 copying PIL\ImageTk.py -> build\lib.win32-2.7 copying PIL\ImageTransform.py -> build\lib.win32-2.7 copying PIL\ImageWin.py -> build\lib.win32-2.7 copying PIL\ImImagePlugin.py -> build\lib.win32-2.7 copying PIL\ImtImagePlugin.py -> build\lib.win32-2.7 copying PIL\IptcImagePlugin.py -> build\lib.win32-2.7 copying PIL\JpegImagePlugin.py -> build\lib.win32-2.7 copying PIL\McIdasImagePlugin.py -> build\lib.win32-2.7 copying PIL\MicImagePlugin.py -> build\lib.win32-2.7 copying PIL\MpegImagePlugin.py -> build\lib.win32-2.7 copying PIL\MspImagePlugin.py -> build\lib.win32-2.7 copying PIL\OleFileIO.py -> build\lib.win32-2.7 copying PIL\PaletteFile.py -> build\lib.win32-2.7 copying PIL\PalmImagePlugin.py -> build\lib.win32-2.7 copying PIL\PcdImagePlugin.py -> build\lib.win32-2.7 copying PIL\PcfFontFile.py -> build\lib.win32-2.7 copying PIL\PcxImagePlugin.py -> build\lib.win32-2.7 copying PIL\PdfImagePlugin.py -> build\lib.win32-2.7 copying PIL\PixarImagePlugin.py -> build\lib.win32-2.7 copying PIL\PngImagePlugin.py -> build\lib.win32-2.7 copying PIL\PpmImagePlugin.py -> build\lib.win32-2.7 copying PIL\PsdImagePlugin.py -> build\lib.win32-2.7 copying PIL\PSDraw.py -> build\lib.win32-2.7 copying PIL\SgiImagePlugin.py -> build\lib.win32-2.7 copying PIL\SpiderImagePlugin.py -> build\lib.win32-2.7 copying PIL\SunImagePlugin.py -> build\lib.win32-2.7 copying PIL\TarIO.py -> build\lib.win32-2.7 copying PIL\TgaImagePlugin.py -> build\lib.win32-2.7 copying PIL\TiffImagePlugin.py -> build\lib.win32-2.7 copying PIL\TiffTags.py -> build\lib.win32-2.7 copying PIL\WalImageFile.py -> build\lib.win32-2.7 copying PIL\WmfImagePlugin.py -> build\lib.win32-2.7 copying PIL\XbmImagePlugin.py -> build\lib.win32-2.7 copying PIL\XpmImagePlugin.py -> build\lib.win32-2.7 copying PIL\XVThumbImagePlugin.py -> build\lib.win32-2.7 copying PIL\__init__.py -> build\lib.win32-2.7 running build_ext Traceback (most recent call last): File "app_main.py", line 51, in run_toplevel File "app_main.py", line 534, in run_it File "", line 1, in File "c:\pypy-1.7\bin\build\PIL\setup.py", line 486, in version=VERSION, File "C:\pypy-1.7\lib-python\modified-2.7\distutils\core.py", line 152, in set up dist.run_commands() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 953, in run _commands self.run_command(cmd) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run _command cmd_obj.run() File "C:\pypy-1.7\site-packages\distribute-0.6.24-py2.7.egg\setuptools\command \install.py", line 53, in run return _install.run(self) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\install.py", line 572, in run self.run_command('build') File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cmd.py", line 326, in run_ command self.distribution.run_command(command) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run _command cmd_obj.run() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\build.py", line 12 7, in run self.run_command(cmd_name) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cmd.py", line 326, in run_ command self.distribution.run_command(command) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\dist.py", line 972, in run _command cmd_obj.run() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\command\build_ext.py", lin e 314, in run force=self.force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\ccompiler.py", line 1054, in new_compiler return klass(None, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", line 309, in __init__ CygwinCCompiler.__init__ (self, verbose, dry_run, force) File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", line 99, in __init__ (status, details) = check_config_h() File "C:\pypy-1.7\lib-python\modified-2.7\distutils\cygwinccompiler.py", line 383, in check_config_h fn = sysconfig.get_config_h_filename() AttributeError: 'module' object has no attribute 'get_config_h_filename' ---------------------------------------- Command C:\pypy-1.7\pypy.exe -c "import setuptools;__file__='c:\\pypy-1.7\\bin\\ build\\PIL\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record c:\user s\jlap\appdata\local\temp\pip-zpbnsd-record\install-record.txt failed with error code 1 Storing complete log in C:\Users\Jlap\AppData\Roaming\pip\pip.log c:\pypy-1.7\bin> From amauryfa at gmail.com Wed Nov 30 00:45:48 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 30 Nov 2011 00:45:48 +0100 Subject: [pypy-dev] PIL, Pypy-1.7, and Win7 In-Reply-To: References: Message-ID: Hi, 2011/11/30 Yaacov Finkelman > Now the pip is working I was trying to use it to install PIL part of > long error message is below. I am using Pypy-1.7 > > Any help? > > The cygwin compiler is not yet supported by pypy C extensions. It's not too difficult though, if you know how to hack in lib-python\modified-2.7\distutils\sysconfig_pypy.py You'll have to implement a get_config_h_filename() similar to the one in sysconfig_cpython.py, but adapted for pypy... And send us your changes when it works! -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed Nov 30 03:47:39 2011 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 Nov 2011 03:47:39 +0100 Subject: [pypy-dev] Optimizing Append Only structures In-Reply-To: References: Message-ID: Hi, On Tue, Nov 29, 2011 at 21:28, Romain Guillebert wrote: > Probably because he (as a clojure developer) likes immutability of data > structures. No, it's really needed for the way it is written: by creating a new dict, the old purefunction results no longer apply. But we are (indeed) using a slightly different approach in PyPy by not copying the dict, but instead creating a new empty 'signature' object that we pass to the purefunction too. We don't have anything exactly similar in PyPy, I think. I would go for something along the lines of: class Cell(object): _immutable_fields_ = ['content?'] # quasi-immutable content = None _all_cells = {} @elidable # same as @purefunction def get_cell(key): return _all_cells.setdefault(key, Cell()) or, depending on the usage, maybe @elidable_promote, if the key should always be a jit-constant. In this way the user gets a Cell corresponding to the key he asked for, and then he can read the 'content' field, which is initially None but may be set to something else. Because it is a quasi-immutable field, this is all done with no machine code produced. If later the same Cell has its 'content' field modified, then the old machine code is discarded and new code is produced. Note the indirection: the JIT should not see the @elidable code, but just call it at tracing time; but the JIT must see the read of the 'content' field, to be able to use the fact that it's a quasi-immutable. A bient?t, Armin. From yeomanyaacov at gmail.com Wed Nov 30 04:27:29 2011 From: yeomanyaacov at gmail.com (Yaacov Finkelman) Date: Tue, 29 Nov 2011 22:27:29 -0500 Subject: [pypy-dev] PIL, Pypy-1.7, and Win7 In-Reply-To: References: Message-ID: so I looked at get_config_h_filename() and it can be ported directly from sysconfig_cpython.py. Fixing this led to a new set of error messages (again below) what's the next step to fix these? Is there a free and easy way to use some other compiler with pypy? Jacob ... C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\i nclude -c _imaging.c -o build\temp.win32-2.7\Release\_imaging.o _imaging.c: In function '_putdata': _imaging.c:1230:13: warning: passing argument 1 of 'PyString_AsString' from incompatible pointer type C:\pypy-1.7\include/pypy_decl.h:339:20: note: expected 'struct PyObject *' b ut argument is of type 'struct PyStringObject *' _imaging.c: In function '_rotate': _imaging.c:1492:5: warning: implicit declaration of function 'fmod' _imaging.c:1492:13: warning: incompatible implicit declaration of built-in f unction 'fmod' _imaging.c: At top level: _imaging.c:3017:5: warning: initialization from incompatible pointer type _imaging.c:3077:5: warning: initialization from incompatible pointer type C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\i nclude -c decode.c -o build\temp.win32-2.7\Release\decode.o C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\i nclude -c encode.c -o build\temp.win32-2.7\Release\encode.o C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\i nclude -c map.c -o build\temp.win32-2.7\Release\map.o In file included from c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../incl ude/winnt.h:192:0, from c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../incl ude/windef.h:253, from c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../incl ude/windows.h:48, from map.c:35: c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:50:21: e rror: duplicate 'signed' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:50:21: e rror: two or more data types in declaration specifiers c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:51:22: e rror: duplicate 'short' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:56:23: e rror: duplicate 'unsigned' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:56:23: e rror: two or more data types in declaration specifiers c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:57:24: e rror: duplicate 'unsigned' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:57:24: e rror: duplicate 'short' error: command 'gcc' failed with exit status 1 Complete output from command C:\pypy-1.7\pypy.exe -c "import setuptools;__fi le__='c:\\pypy-1.7\\bin\\build\\PIL\\setup.py';exec(compile(open(__file__).read( ).replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally -managed --record c:\users\jlap\appdata\local\temp\pip-dmwurk-record\install-rec ord.txt: WARNING: '' not a valid package name; please use only.-separated package nam es in setup.py running install running build running build_py copying PIL\ArgImagePlugin.py -> build\lib.win32-2.7 copying PIL\BdfFontFile.py -> build\lib.win32-2.7 copying PIL\BmpImagePlugin.py -> build\lib.win32-2.7 copying PIL\BufrStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\ContainerIO.py -> build\lib.win32-2.7 copying PIL\CurImagePlugin.py -> build\lib.win32-2.7 copying PIL\DcxImagePlugin.py -> build\lib.win32-2.7 copying PIL\EpsImagePlugin.py -> build\lib.win32-2.7 copying PIL\ExifTags.py -> build\lib.win32-2.7 copying PIL\FitsStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\FliImagePlugin.py -> build\lib.win32-2.7 copying PIL\FontFile.py -> build\lib.win32-2.7 copying PIL\FpxImagePlugin.py -> build\lib.win32-2.7 copying PIL\GbrImagePlugin.py -> build\lib.win32-2.7 copying PIL\GdImageFile.py -> build\lib.win32-2.7 copying PIL\GifImagePlugin.py -> build\lib.win32-2.7 copying PIL\GimpGradientFile.py -> build\lib.win32-2.7 copying PIL\GimpPaletteFile.py -> build\lib.win32-2.7 copying PIL\GribStubImagePlugin.py -> build\lib.win32-2.7 copying PIL\Hdf5StubImagePlugin.py -> build\lib.win32-2.7 copying PIL\IcnsImagePlugin.py -> build\lib.win32-2.7 copying PIL\IcoImagePlugin.py -> build\lib.win32-2.7 copying PIL\Image.py -> build\lib.win32-2.7 copying PIL\ImageChops.py -> build\lib.win32-2.7 copying PIL\ImageCms.py -> build\lib.win32-2.7 copying PIL\ImageColor.py -> build\lib.win32-2.7 copying PIL\ImageDraw.py -> build\lib.win32-2.7 copying PIL\ImageDraw2.py -> build\lib.win32-2.7 copying PIL\ImageEnhance.py -> build\lib.win32-2.7 copying PIL\ImageFile.py -> build\lib.win32-2.7 copying PIL\ImageFileIO.py -> build\lib.win32-2.7 copying PIL\ImageFilter.py -> build\lib.win32-2.7 copying PIL\ImageFont.py -> build\lib.win32-2.7 copying PIL\ImageGL.py -> build\lib.win32-2.7 copying PIL\ImageGrab.py -> build\lib.win32-2.7 copying PIL\ImageMath.py -> build\lib.win32-2.7 copying PIL\ImageMode.py -> build\lib.win32-2.7 copying PIL\ImageOps.py -> build\lib.win32-2.7 copying PIL\ImagePalette.py -> build\lib.win32-2.7 copying PIL\ImagePath.py -> build\lib.win32-2.7 copying PIL\ImageQt.py -> build\lib.win32-2.7 copying PIL\ImageSequence.py -> build\lib.win32-2.7 copying PIL\ImageShow.py -> build\lib.win32-2.7 copying PIL\ImageStat.py -> build\lib.win32-2.7 copying PIL\ImageTk.py -> build\lib.win32-2.7 copying PIL\ImageTransform.py -> build\lib.win32-2.7 copying PIL\ImageWin.py -> build\lib.win32-2.7 copying PIL\ImImagePlugin.py -> build\lib.win32-2.7 copying PIL\ImtImagePlugin.py -> build\lib.win32-2.7 copying PIL\IptcImagePlugin.py -> build\lib.win32-2.7 copying PIL\JpegImagePlugin.py -> build\lib.win32-2.7 copying PIL\McIdasImagePlugin.py -> build\lib.win32-2.7 copying PIL\MicImagePlugin.py -> build\lib.win32-2.7 copying PIL\MpegImagePlugin.py -> build\lib.win32-2.7 copying PIL\MspImagePlugin.py -> build\lib.win32-2.7 copying PIL\OleFileIO.py -> build\lib.win32-2.7 copying PIL\PaletteFile.py -> build\lib.win32-2.7 copying PIL\PalmImagePlugin.py -> build\lib.win32-2.7 copying PIL\PcdImagePlugin.py -> build\lib.win32-2.7 copying PIL\PcfFontFile.py -> build\lib.win32-2.7 copying PIL\PcxImagePlugin.py -> build\lib.win32-2.7 copying PIL\PdfImagePlugin.py -> build\lib.win32-2.7 copying PIL\PixarImagePlugin.py -> build\lib.win32-2.7 copying PIL\PngImagePlugin.py -> build\lib.win32-2.7 copying PIL\PpmImagePlugin.py -> build\lib.win32-2.7 copying PIL\PsdImagePlugin.py -> build\lib.win32-2.7 copying PIL\PSDraw.py -> build\lib.win32-2.7 copying PIL\SgiImagePlugin.py -> build\lib.win32-2.7 copying PIL\SpiderImagePlugin.py -> build\lib.win32-2.7 copying PIL\SunImagePlugin.py -> build\lib.win32-2.7 copying PIL\TarIO.py -> build\lib.win32-2.7 copying PIL\TgaImagePlugin.py -> build\lib.win32-2.7 copying PIL\TiffImagePlugin.py -> build\lib.win32-2.7 copying PIL\TiffTags.py -> build\lib.win32-2.7 copying PIL\WalImageFile.py -> build\lib.win32-2.7 copying PIL\WmfImagePlugin.py -> build\lib.win32-2.7 copying PIL\XbmImagePlugin.py -> build\lib.win32-2.7 copying PIL\XpmImagePlugin.py -> build\lib.win32-2.7 copying PIL\XVThumbImagePlugin.py -> build\lib.win32-2.7 copying PIL\__init__.py -> build\lib.win32-2.7 running build_ext warning: Python's pyconfig.h doesn't seem to support your compiler. Reason: 'C:\ pypy-1.7\include\pyconfig.h' does not mention '__GNUC__'. Compiling may fail bec ause of undefined preprocessor macros. building '_imaging' extension C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\inclu de -c _imaging.c -o build\temp.win32-2.7\Release\_imaging.o _imaging.c: In function '_putdata': _imaging.c:1230:13: warning: passing argument 1 of 'PyString_AsString' from inco mpatible pointer type C:\pypy-1.7\include/pypy_decl.h:339:20: note: expected 'struct PyObject *' but a rgument is of type 'struct PyStringObject *' _imaging.c: In function '_rotate': _imaging.c:1492:5: warning: implicit declaration of function 'fmod' _imaging.c:1492:13: warning: incompatible implicit declaration of built-in funct ion 'fmod' _imaging.c: At top level: _imaging.c:3017:5: warning: initialization from incompatible pointer type _imaging.c:3077:5: warning: initialization from incompatible pointer type C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\inclu de -c decode.c -o build\temp.win32-2.7\Release\decode.o C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\inclu de -c encode.c -o build\temp.win32-2.7\Release\encode.o C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IlibImaging -IC:\pypy-1.7\inclu de -c map.c -o build\temp.win32-2.7\Release\map.o In file included from c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/ winnt.h:192:0, from c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/ windef.h:253, from c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/ windows.h:48, from map.c:35: c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:50:21: error : duplicate 'signed' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:50:21: error : two or more data types in declaration specifiers c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:51:22: error : duplicate 'short' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:56:23: error : duplicate 'unsigned' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:56:23: error : two or more data types in declaration specifiers c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:57:24: error : duplicate 'unsigned' c:\mingw\bin\../lib/gcc/mingw32/4.5.2/../../../../include/basetsd.h:57:24: error : duplicate 'short' error: command 'gcc' failed with exit status 1 ---------------------------------------- Command C:\pypy-1.7\pypy.exe -c "import setuptools;__file__='c:\\pypy-1.7\\bin\\ build\\PIL\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record c:\user s\jlap\appdata\local\temp\pip-dmwurk-record\install-record.txt failed with error code 1 Storing complete log in C:\Users\Jlap\AppData\Roaming\pip\pip.log c:\pypy-1.7\bin> On Tue, Nov 29, 2011 at 6:45 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/11/30 Yaacov Finkelman >> >> Now the pip is working I was trying to use it to install PIL part of >> long error message is below. I am using Pypy-1.7 >> >> Any help? >> > > The cygwin compiler is not yet supported by pypy C extensions. > > It's not too difficult though, if you know how to hack in > lib-python\modified-2.7\distutils\sysconfig_pypy.py > You'll have to implement a?get_config_h_filename() > similar to the one in sysconfig_cpython.py, but adapted for pypy... > And send us your changes when it works! > -- > Amaury Forgeot d'Arc > From santagada at gmail.com Wed Nov 30 17:19:47 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 30 Nov 2011 14:19:47 -0200 Subject: [pypy-dev] Pip, Pypy, and Win7 In-Reply-To: References: Message-ID: On Tue, Nov 29, 2011 at 8:53 PM, Yaacov Finkelman wrote: > Thank You! > > pip.exe and pip-2.7.exe are indeed in a subfolder called bin. Can we > add that to the new version of the docs? > > When should I be using pip vs. using pip-2.7? Doesn't matter, both will use pypy-1.7. The two binaries is in the case you have many python versions and whant to use the specific pip of one then you use the one with the version in the end. -- Leonardo Santagada