From matti.picus at gmail.com Tue Nov 1 12:10:10 2016 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 1 Nov 2016 18:10:10 +0200 Subject: [pypy-dev] Colo in Germany for the Windows build slave? In-Reply-To: References: Message-ID: On 25/10/16 22:49, Yury V. Zaytsev wrote: > Hi there, > > I've been running the Windows build slave for the last ~1.5 years, but > all good things come to an end, and I've been informed that I must > shutdown the machine by the end of the year (that was the bad news). > > The "good" news is that I've been promised that I can have the > hardware, so, in theory, if I can find colo to host it (Germany, > preferably Cologne area), I can replace the hard drives, reshuffle the > RAM, reinstall the system and plug it back in. If memory serves me > well, it's an old Dell PowerEdge R710 rack mount server (2U). > > It used to run other VMs unrelated to the PyPy project, but I only > plan to use it to host build VMs for OSS projects that I'm involved in > after "decommissioning". Any takers? I think it would be best to talk > via private email to avoid annoying list subscribers. > Hey Yury. Thanks for running the win32 buildbot. When you say "colo to host it", what are you actually looking for? Electricity, airconditioning, and internet connection? This could be someone's office or private residence or are you thinking of some kind of hosting service? Not that I live anywhere near Cologne, but maybe better defining the request might help tease out a response here... Matti From njs at pobox.com Tue Nov 1 12:53:19 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Nov 2016 09:53:19 -0700 Subject: [pypy-dev] Colo in Germany for the Windows build slave? In-Reply-To: References: Message-ID: If there's a need for win32 hosting then it's also possible that someone like Microsoft would be willing to donate time on azure, which might be simpler than trying to work out colo stuff. On Nov 1, 2016 9:10 AM, "Matti Picus" wrote: > On 25/10/16 22:49, Yury V. Zaytsev wrote: > > Hi there, >> >> I've been running the Windows build slave for the last ~1.5 years, but >> all good things come to an end, and I've been informed that I must shutdown >> the machine by the end of the year (that was the bad news). >> >> The "good" news is that I've been promised that I can have the hardware, >> so, in theory, if I can find colo to host it (Germany, preferably Cologne >> area), I can replace the hard drives, reshuffle the RAM, reinstall the >> system and plug it back in. If memory serves me well, it's an old Dell >> PowerEdge R710 rack mount server (2U). >> >> It used to run other VMs unrelated to the PyPy project, but I only plan >> to use it to host build VMs for OSS projects that I'm involved in after >> "decommissioning". Any takers? I think it would be best to talk via private >> email to avoid annoying list subscribers. >> >> Hey Yury. Thanks for running the win32 buildbot. When you say "colo to > host it", what are you actually looking for? Electricity, airconditioning, > and internet connection? > This could be someone's office or private residence or are you thinking of > some kind of hosting service? > Not that I live anywhere near Cologne, but maybe better defining the > request might help tease out a response here... > Matti > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Nov 1 12:58:53 2016 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 1 Nov 2016 18:58:53 +0200 Subject: [pypy-dev] Colo in Germany for the Windows build slave? In-Reply-To: References: Message-ID: Hi Yury I don't have any particularly good answers, but thanks a lot for doing that! Getting buildbots *maintained* is surprisingly tedious job that only few succeed at. Getting that done on windows is double hard, we're all super thankful! On Tue, Oct 25, 2016 at 9:49 PM, Yury V. Zaytsev wrote: > Hi there, > > I've been running the Windows build slave for the last ~1.5 years, but all > good things come to an end, and I've been informed that I must shutdown the > machine by the end of the year (that was the bad news). > > The "good" news is that I've been promised that I can have the hardware, so, > in theory, if I can find colo to host it (Germany, preferably Cologne area), > I can replace the hard drives, reshuffle the RAM, reinstall the system and > plug it back in. If memory serves me well, it's an old Dell PowerEdge R710 > rack mount server (2U). > > It used to run other VMs unrelated to the PyPy project, but I only plan to > use it to host build VMs for OSS projects that I'm involved in after > "decommissioning". Any takers? I think it would be best to talk via private > email to avoid annoying list subscribers. > > -- > Sincerely yours, > Yury V. Zaytsev > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From yury at shurup.com Tue Nov 1 13:11:36 2016 From: yury at shurup.com (Yury V. Zaytsev) Date: Tue, 1 Nov 2016 18:11:36 +0100 (CET) Subject: [pypy-dev] Colo in Germany for the Windows build slave? In-Reply-To: References: Message-ID: On Tue, 1 Nov 2016, Nathaniel Smith wrote: > If there's a need for win32 hosting then it's also possible that someone > like Microsoft would be willing to donate time on azure, which might be > simpler than trying to work out colo stuff. Hi Nathaniel, In my personal opinion, this is exactly where companies like Microsoft have to step in: what does it cost them to give PyPy project a few free Azure VMs, and isn't it in their best interest after all? But you know what, the paradox is that they don't seem to care much, whereas I'm spending my private time to make sure PyPy runs on their platform, and I'm not even otherwise using Windows, be it privately or professionally, and generally try to stay away from their products... Having that said, I been trying to reach out to Steve Dower privately several times to get something going, but he's never got back to me; I thought he is the most appropriate person to talk to, and I don't have any contacts at Microsoft anyways. If you can connect me with someone at MS who has the power to make VMs happen, I'd be happy to go for it, instead of doing it the hard way... but so far, I've had no luck with it. -- Sincerely yours, Yury V. Zaytsev From yury at shurup.com Tue Nov 1 13:20:35 2016 From: yury at shurup.com (Yury V. Zaytsev) Date: Tue, 1 Nov 2016 18:20:35 +0100 (CET) Subject: [pypy-dev] Colo in Germany for the Windows build slave? In-Reply-To: References: Message-ID: Hi Matti, > Not that I live anywhere near Cologne, but maybe better defining the > request might help tease out a response here... You are right, maybe speaking sysadmin lingo wasn't the smartest idea on my part, but then again, not sure anyone who's not using the word "colo" everyday is going to be able to help... > When you say "colo to host it", what are you actually looking for? > Electricity, airconditioning, and internet connection? I'm looking for 2U of rack space supplied with electricity and conditioned air, and 2 network connections, one to the Internet and one to a private LAN for out of band management console; I might get the rails as well, but not sure about that, and that's often mount specific anyways. > This could be someone's office or private residence or are you thinking > of some kind of hosting service? Usually, you don't get full size air conditioned racks at private residences, and most offices have only telco racks, but then again, if I had a house, I would have certainly installed a rack in the basement ;-) -- Sincerely yours, Yury V. Zaytsev From armin.rigo at gmail.com Wed Nov 2 04:38:00 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 2 Nov 2016 09:38:00 +0100 Subject: [pypy-dev] SSL module Py3.5, Github? Bitbucket? In-Reply-To: References: Message-ID: Hi Richard, On 31 October 2016 at 16:20, Richard Plangger wrote: > I'm unsure what the current rule for new project is. Github? Bitbucket? > > VMProf was initiated on Github, what did we learn from that experiment? > Is the bar lower for new contributions? +1 for Github. (I personally prefer hg over git but I think hggit is good enough for most purposes.) A bient?t, Armin. From wouter.pypy at richtlijn.be Wed Nov 2 04:48:15 2016 From: wouter.pypy at richtlijn.be (Wouter van Heijst) Date: Wed, 2 Nov 2016 08:48:15 +0000 Subject: [pypy-dev] Colo in Germany for the Windows build slave? In-Reply-To: References: Message-ID: <20161102084815.GA27282@richtlijn.be> On Tue, Nov 01, 2016 at 18:20:35 +0100, Yury V. Zaytsev wrote: > Hi Matti, > > >Not that I live anywhere near Cologne, but maybe better defining the > >request might help tease out a response here... > > You are right, maybe speaking sysadmin lingo wasn't the smartest idea on my > part, but then again, not sure anyone who's not using the word "colo" > everyday is going to be able to help... > > >When you say "colo to host it", what are you actually looking for? > >Electricity, airconditioning, and internet connection? > > I'm looking for 2U of rack space supplied with electricity and conditioned > air, and 2 network connections, one to the Internet and one to a private LAN > for out of band management console; I might get the rails as well, but not > sure about that, and that's often mount specific anyways. Within Cologne I'd start with contacting Digitale Kultur e.V. (http://www.digitalekultur.org/en/) or Chaos Computer Club Cologne e.V. (https://koeln.ccc.de/). Both have members familiar with hosting. Outside of Cologne I'd try something like Hetzner. Bit more than needed, but 1/3rd of a rack goes for 119?/month. https://www.hetzner.de/de/hosting/colocation_rack/drittel I'll ask some of my ex-colleagues if their MS contacts can help out. HTH, Wouter From armin.rigo at gmail.com Wed Nov 2 04:57:32 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 2 Nov 2016 09:57:32 +0100 Subject: [pypy-dev] Colo in Germany for the Windows build slave? In-Reply-To: References: Message-ID: Hi Yury, On 1 November 2016 at 18:20, Yury V. Zaytsev wrote: > You are right, maybe speaking sysadmin lingo wasn't the smartest idea on my > part, but then again, not sure anyone who's not using the word "colo" > everyday is going to be able to help... Fwiw, I guess you mean https://en.wikipedia.org/wiki/Colocation_centre . As Matti points out, not all buildbots are necessarily rack-mounted hardware. Some of them are just regular machines that happen to be otherwise mostly idle, that somebody leaves running. We do however also have rack-mounted machines in business/university settings. In practice the first kind of hardware tends to silently disappear after a few months, with a few exceptions. So I guess a colo looks like a good plan. A bient?t, Armin. From armin.rigo at gmail.com Wed Nov 2 05:18:40 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 2 Nov 2016 10:18:40 +0100 Subject: [pypy-dev] PGO Optimized Binary In-Reply-To: <0151F66FF725AC42A760DA612754C5F8198E093B@ORSMSX104.amr.corp.intel.com> References: <0151F66FF725AC42A760DA612754C5F8198E093B@ORSMSX104.amr.corp.intel.com> Message-ID: Hi, On 31 October 2016 at 22:28, Singh, Yashwardhan wrote: > We applied compiler assisted optimization technique called PGO or Profile Guided Optimization while building PyPy, and found performance got improved by up to 22.4% on the Grand Unified Python Benchmark (GUPB) from ?hg clone https://hg.python.org/benchmarks?. The below result table shows majority of 51 micros got performance boost with 8 got performance regression. The kind of performance improvement you are measuring involves only short- or very short-running programs. A few years ago we'd have shrugged it off as irrelevant---"please modify the benchmarks so that they run for at least 10 seconds, more if they are larger"---because the JIT compiler doesn't have a chance to warm up. But we'd also have shrugged off your whole attempt---"PGO optimization cannot change anything to the speed of JIT-produced machine code". Nowadays we tend to look more seriously at the cold or warming-up performance too, or at least we know that we should look there. There are (stalled) plans of setting up a second benchmark suite for PyPy which focuses on this. You can get an estimate of whether you're looking at cold or hot code: compare the timings with CPython. Also, you can set the environment variable ``PYPYLOG=jit-summary:-`` and look at the first 2 lines to see how much time was spent warming up the JIT (or attempting to). Note that we did enable PGO long ago, with modest benefits. We gave up when our JIT compiler became good enough. Maybe now is the time to try again (and also, PGO itself might have improved in the meantime). > We?d like to get some input on how to contribute our optimization recipe to the PyPy dev tree, perhaps by creating an item to the PyPy issue tracker? The best would be to create a pull request so that we can look at your changes more easily. > In addition, we would also appreciate any other benchmark or real world use based workload as alternatives to evaluate this. You can take any Python program that runs either very shortly or not faster than CPython. For a larger example (with Python 2.7): cd rpython/jit/tl python ../../bin/rpython -O2 --source targettlr # 24 secs pypy ../../bin/rpython -O2 --source targettlr # 39 secs A bient?t, Armin. From armin.rigo at gmail.com Wed Nov 2 06:41:32 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 2 Nov 2016 11:41:32 +0100 Subject: [pypy-dev] pypy-config In-Reply-To: References: <2192620.DiRuYLhWm2@dragon.local> <4d5a8a3b-2a67-3b17-58fa-00695a111071@gmail.com> Message-ID: Hi, On 20 October 2016 at 21:18, Yury V. Zaytsev wrote: >> AFAICT, python-config is provided by the downstream package maintainer. >> For instance, in debian it is provided by their python-dev package. Since it >> is not an integral part of python, I'm not sure it should be an integral >> part of pypy, but it is trivial to copy-and-modify. > > FYI, this used to be the case a long while ago, but python-config has been > integrated into Python 2.5 since ~2006: > > https://mail.python.org/pipermail/patches/2006-April/019478.html I can only recommend that someone submits a pull request for it, or at least a comprehensive description of when it is used and what it should answer (ideally as a series of tests). I don't have any good way to test the result myself and I don't want to guess. A bient?t, Armin. From matti.picus at gmail.com Wed Nov 2 08:57:34 2016 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 2 Nov 2016 14:57:34 +0200 Subject: [pypy-dev] pypy-config In-Reply-To: References: <2192620.DiRuYLhWm2@dragon.local> <4d5a8a3b-2a67-3b17-58fa-00695a111071@gmail.com> Message-ID: On 02/11/16 12:41, Armin Rigo wrote: > Hi, > > On 20 October 2016 at 21:18, Yury V. Zaytsev wrote: >>> AFAICT, python-config is provided by the downstream package maintainer. >>> For instance, in debian it is provided by their python-dev package. Since it >>> is not an integral part of python, I'm not sure it should be an integral >>> part of pypy, but it is trivial to copy-and-modify. >> FYI, this used to be the case a long while ago, but python-config has been >> integrated into Python 2.5 since ~2006: >> >> https://mail.python.org/pipermail/patches/2006-April/019478.html > I can only recommend that someone submits a pull request for it, or at > least a comprehensive description of when it is used and what it > should answer (ideally as a series of tests). I don't have any good > way to test the result myself and I don't want to guess. > > > A bient?t, > > Armin. I started work on this on the pypy-config branch, I copied the python-config script from CPython. The script itself lives in the top-level PyPy directory on that branch and can be run with "--help" to see the options Tests are definitely needed, optimally the tests would use the flags from the script to try to compile some C code. As far as real-world use cases, PyQt uses this script. Of course, fixing this will not magically make PyQt work, we need to finish the missing-tp_new branch and a way to override tp_alloc. Matti From matti.picus at gmail.com Wed Nov 2 11:11:00 2016 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 2 Nov 2016 17:11:00 +0200 Subject: [pypy-dev] starting a release cycle Message-ID: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> I am thinking of pushing out what I guess should be PyPy2.7-5.6.0, we released PyPy2.7-v5.4.0 on Aug 31 and PyPy3.3-v5.5.0 on Oct 12. Are there outstanding branches that just have to be in this release or issues that we consider blockers? Would someone else like to pair with me for the release so they could do the next one? FWIW, I tick off the boxes on http://pypy.readthedocs.io/en/latest/how-to-release.html, not too hard really. I started a release notice http://pypy.readthedocs.io/en/latest/release-pypy2.7-v5.6.0.html, edits are encouraged. It is a bit dry/formal, for what is quite a leap forward in terms of stdlib2.7.12, cpyext, OpenSSL, ... Matti From wlavrijsen at lbl.gov Wed Nov 2 11:41:04 2016 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 2 Nov 2016 08:41:04 -0700 (PDT) Subject: [pypy-dev] starting a release cycle In-Reply-To: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> References: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> Message-ID: Matti, On Wednesday 2016-11-02 17:11, Matti Picus wrote: > Are there outstanding branches that just have to be in this release or > issues that we consider blockers? I want at some point in the near future push cling-support onto master. I assume there are no objections (if there are, please speak up :) ). What would be the most convenient time (need not be this release, in fact right after, to get maximum lead time for the next, is fine by me)? Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From planrichi at gmail.com Wed Nov 2 11:42:01 2016 From: planrichi at gmail.com (Richard Plangger) Date: Wed, 2 Nov 2016 16:42:01 +0100 Subject: [pypy-dev] starting a release cycle In-Reply-To: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> References: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> Message-ID: Hi, > Would someone else like to pair with me for the release so they could do > the next one? Yes, I would like to help out here! In cape town I have done the same release procedure for pypy3 (Python 3.3). It annoyed me :) I think most of the steps could be done automatically (ideally a script that is started on a server and runs to completion there, passing through all steps from kicking buildbot, downloading, repackaging, uploading to bitbucket, generating checksums, ...). @mattip what do you think? Would that make sense to try to automate that? > I started a release notice > http://pypy.readthedocs.io/en/latest/release-pypy2.7-v5.6.0.html, edits > are encouraged. I'll edit it as soon as my branch is merged! Cheers, Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From matti.picus at gmail.com Wed Nov 2 12:27:26 2016 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 2 Nov 2016 18:27:26 +0200 Subject: [pypy-dev] starting a release cycle In-Reply-To: References: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> Message-ID: On 02/11/16 17:42, Richard Plangger wrote: > Hi, > >> Would someone else like to pair with me for the release so they could do >> the next one? > Yes, I would like to help out here! In cape town I have done the same > release procedure for pypy3 (Python 3.3). It annoyed me :) > > I think most of the steps could be done automatically (ideally a script > that is started on a server and runs to completion there, passing > through all steps from kicking buildbot, downloading, repackaging, > uploading to bitbucket, generating checksums, ...). @mattip what do you > think? Would that make sense to try to automate that? There is are the force-builds.py and repackage.sh scripts in pypy/tool/release, I just edit repackage.sh by hand before running. AFAIK there is no API to bitbucket for uploading. We could maybe write something to automate the edit of pypy.org/source/download.txt I don't know about the trade-off of where automation maintenance becomes more expensive than just doing it by hand Everyone has their own break even point Matti From cfbolz at gmx.de Wed Nov 2 12:49:16 2016 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 02 Nov 2016 17:49:16 +0100 Subject: [pypy-dev] starting a release cycle In-Reply-To: References: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> Message-ID: <4a1c3907-8811-4326-94df-8c419160b137@email.android.com> FWIW, there is this: https://bitbucket.org/okusche/bitbucket-curl-upload-to-repo-downloads Which I haven't tested, but the author says works. Carl Friedrich On November 2, 2016 5:27:26 PM GMT+01:00, Matti Picus wrote: >On 02/11/16 17:42, Richard Plangger wrote: > >> Hi, >> >>> Would someone else like to pair with me for the release so they >could do >>> the next one? >> Yes, I would like to help out here! In cape town I have done the same >> release procedure for pypy3 (Python 3.3). It annoyed me :) >> >> I think most of the steps could be done automatically (ideally a >script >> that is started on a server and runs to completion there, passing >> through all steps from kicking buildbot, downloading, repackaging, >> uploading to bitbucket, generating checksums, ...). @mattip what do >you >> think? Would that make sense to try to automate that? > >There is are the force-builds.py and repackage.sh scripts in >pypy/tool/release, I just edit repackage.sh by hand before running. >AFAIK there is no API to bitbucket for uploading. >We could maybe write something to automate the edit of >pypy.org/source/download.txt >I don't know about the trade-off of where automation maintenance >becomes >more expensive than just doing it by hand >Everyone has their own break even point >Matti >_______________________________________________ >pypy-dev mailing list >pypy-dev at python.org >https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Thu Nov 3 13:21:23 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Thu, 3 Nov 2016 18:21:23 +0100 Subject: [pypy-dev] starting a release cycle In-Reply-To: <4a1c3907-8811-4326-94df-8c419160b137@email.android.com> References: <2ef45c1e-aa02-f110-1e55-79eb194b1a89@gmail.com> <4a1c3907-8811-4326-94df-8c419160b137@email.android.com> Message-ID: Hi, On 2 November 2016 at 17:49, Carl Friedrich Bolz wrote: > FWIW, there is this: > > https://bitbucket.org/okusche/bitbucket-curl-upload-to-repo-downloads > > Which I haven't tested, but the author says works. I can continue to volunteer to download the nightly builds and re-upload them to bitbucket manually (it's really one operation, not even file-by-file). Overall I agree with matti that trying to automate the whole process may be pointless and leads to more work trying to keep the scripts up-to-date, instead of just following the steps in a text file (as long as the number of steps is a small number). A bient?t, Armin. From dannym at scratchpost.org Fri Nov 4 07:09:12 2016 From: dannym at scratchpost.org (Danny Milosavljevic) Date: Fri, 4 Nov 2016 12:09:12 +0100 Subject: [pypy-dev] pypy3.3: building extensions: environment variables CC etc - not honored In-Reply-To: <2d72285c-6456-1a9b-cf6d-f9858f43bb4f@gmx.de> References: <20161008143911.100bd295@scratchpost.org> <2d72285c-6456-1a9b-cf6d-f9858f43bb4f@gmx.de> Message-ID: <20161104120912.72d468c0@scratchpost.org> Hi, I've created - but when I press the Attach button nothing happens (Javascript console says nothing either). Therefore, I was not able to attach the patch. Sorry. From yashwardhan.singh at intel.com Tue Nov 8 17:30:55 2016 From: yashwardhan.singh at intel.com (Singh, Yashwardhan) Date: Tue, 8 Nov 2016 22:30:55 +0000 Subject: [pypy-dev] PGO Optimized Binary In-Reply-To: References: <0151F66FF725AC42A760DA612754C5F8198E093B@ORSMSX104.amr.corp.intel.com>, Message-ID: <0151F66FF725AC42A760DA612754C5F8198E161F@ORSMSX104.amr.corp.intel.com> Hi Armin, Thanks for your feedback. We ran one of the program suggested by you as an example for evaluation: cd rpython/jit/tl non-pgo-pypy ../../bin/rpython -O2 --source targettlr pgo-pypy ../../bin/rpython -O2 --source targettlr We got the following results : Non-Pgo pypy - [Timer] Timings: [Timer] annotate --- 7.5 s [Timer] rtype_lltype --- 5.8 s [Timer] backendopt_lltype --- 3.6 s [Timer] stackcheckinsertion_lltype --- 0.1 s [Timer] database_c --- 19.6 s [Timer] source_c --- 2.6 s [Timer] ========================================= [Timer] Total: --- 39.2 s PGO-pypy : [Timer] Timings: [Timer] annotate --- 7.6 s [Timer] rtype_lltype --- 5.1 s [Timer] backendopt_lltype --- 3.1 s [Timer] stackcheckinsertion_lltype --- 0.0 s [Timer] database_c --- 18.5 s [Timer] source_c --- 2.3 s [Timer] ========================================= [Timer] Total: --- 36.6 s The delta in performance between these two is about 8%. We are working on getting the data to identify the % of interpreted code vs the jited code for both the binaries. We are also working on creating a pull request to get a better feedback on the change. Regards Yash ________________________________________ From: Armin Rigo [armin.rigo at gmail.com] Sent: Wednesday, November 02, 2016 2:18 AM To: Singh, Yashwardhan Cc: pypy-dev at python.org Subject: Re: [pypy-dev] PGO Optimized Binary Hi, On 31 October 2016 at 22:28, Singh, Yashwardhan wrote: > We applied compiler assisted optimization technique called PGO or Profile Guided Optimization while building PyPy, and found performance got improved by up to 22.4% on the Grand Unified Python Benchmark (GUPB) from ?hg clone https://hg.python.org/benchmarks?. The below result table shows majority of 51 micros got performance boost with 8 got performance regression. The kind of performance improvement you are measuring involves only short- or very short-running programs. A few years ago we'd have shrugged it off as irrelevant---"please modify the benchmarks so that they run for at least 10 seconds, more if they are larger"---because the JIT compiler doesn't have a chance to warm up. But we'd also have shrugged off your whole attempt---"PGO optimization cannot change anything to the speed of JIT-produced machine code". Nowadays we tend to look more seriously at the cold or warming-up performance too, or at least we know that we should look there. There are (stalled) plans of setting up a second benchmark suite for PyPy which focuses on this. You can get an estimate of whether you're looking at cold or hot code: compare the timings with CPython. Also, you can set the environment variable ``PYPYLOG=jit-summary:-`` and look at the first 2 lines to see how much time was spent warming up the JIT (or attempting to). Note that we did enable PGO long ago, with modest benefits. We gave up when our JIT compiler became good enough. Maybe now is the time to try again (and also, PGO itself might have improved in the meantime). > We?d like to get some input on how to contribute our optimization recipe to the PyPy dev tree, perhaps by creating an item to the PyPy issue tracker? The best would be to create a pull request so that we can look at your changes more easily. > In addition, we would also appreciate any other benchmark or real world use based workload as alternatives to evaluate this. You can take any Python program that runs either very shortly or not faster than CPython. For a larger example (with Python 2.7): cd rpython/jit/tl python ../../bin/rpython -O2 --source targettlr # 24 secs pypy ../../bin/rpython -O2 --source targettlr # 39 secs A bient?t, Armin. From matti.picus at gmail.com Wed Nov 9 15:42:13 2016 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 9 Nov 2016 22:42:13 +0200 Subject: [pypy-dev] PyPy2.7-v5.6 pre-release builds available for testing Message-ID: <868b76f5-c57f-2b5f-1a2e-a4ccf9a5ed71@gmail.com> The buildbot runs on the release branch seem reasonably green, we are getting closer to a release. Please download the tarballs available here http://buildbot.pypy.org/nightly/release-pypy2.7-5.x and try them out with your favorite frameworks. The release notes available here http://pypy.readthedocs.io/en/latest/release-pypy2.7-v5.6.0.html are waiting for your suggestions. If there are no complaints, we will begin uploading the correctly packaged tarballs to the official download site over the next few days. Matti and Richard From fijall at gmail.com Thu Nov 10 13:42:37 2016 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 10 Nov 2016 20:42:37 +0200 Subject: [pypy-dev] PGO Optimized Binary In-Reply-To: <0151F66FF725AC42A760DA612754C5F8198E161F@ORSMSX104.amr.corp.intel.com> References: <0151F66FF725AC42A760DA612754C5F8198E093B@ORSMSX104.amr.corp.intel.com> <0151F66FF725AC42A760DA612754C5F8198E161F@ORSMSX104.amr.corp.intel.com> Message-ID: Hi 8% of that is very good if you can reproduce it across multiple runs (there is a pretty high variance I would think). You can also try running with --jit off. This gives you an indication of the speed of interpreter, which is a part of warmup On Wed, Nov 9, 2016 at 12:30 AM, Singh, Yashwardhan wrote: > Hi Armin, > > > Thanks for your feedback. > We ran one of the program suggested by you as an example for evaluation: > cd rpython/jit/tl > non-pgo-pypy ../../bin/rpython -O2 --source targettlr > pgo-pypy ../../bin/rpython -O2 --source targettlr > > We got the following results : > Non-Pgo pypy - > [Timer] Timings: > [Timer] annotate --- 7.5 s > [Timer] rtype_lltype --- 5.8 s > [Timer] backendopt_lltype --- 3.6 s > [Timer] stackcheckinsertion_lltype --- 0.1 s > [Timer] database_c --- 19.6 s > [Timer] source_c --- 2.6 s > [Timer] ========================================= > [Timer] Total: --- 39.2 s > > PGO-pypy : > [Timer] Timings: > [Timer] annotate --- 7.6 s > [Timer] rtype_lltype --- 5.1 s > [Timer] backendopt_lltype --- 3.1 s > [Timer] stackcheckinsertion_lltype --- 0.0 s > [Timer] database_c --- 18.5 s > [Timer] source_c --- 2.3 s > [Timer] ========================================= > [Timer] Total: --- 36.6 s > > The delta in performance between these two is about 8%. > > We are working on getting the data to identify the % of interpreted code vs the jited code for both the binaries. We are also working on creating a pull request to get a better feedback on the change. > > Regards > Yash > > ________________________________________ > From: Armin Rigo [armin.rigo at gmail.com] > Sent: Wednesday, November 02, 2016 2:18 AM > To: Singh, Yashwardhan > Cc: pypy-dev at python.org > Subject: Re: [pypy-dev] PGO Optimized Binary > > Hi, > > On 31 October 2016 at 22:28, Singh, Yashwardhan > wrote: >> We applied compiler assisted optimization technique called PGO or Profile Guided Optimization while building PyPy, and found performance got improved by up to 22.4% on the Grand Unified Python Benchmark (GUPB) from ?hg clone https://hg.python.org/benchmarks?. The below result table shows majority of 51 micros got performance boost with 8 got performance regression. > > The kind of performance improvement you are measuring involves only > short- or very short-running programs. A few years ago we'd have > shrugged it off as irrelevant---"please modify the benchmarks so that > they run for at least 10 seconds, more if they are larger"---because > the JIT compiler doesn't have a chance to warm up. But we'd also have > shrugged off your whole attempt---"PGO optimization cannot change > anything to the speed of JIT-produced machine code". > > Nowadays we tend to look more seriously at the cold or warming-up > performance too, or at least we know that we should look there. There > are (stalled) plans of setting up a second benchmark suite for PyPy > which focuses on this. > > You can get an estimate of whether you're looking at cold or hot code: > compare the timings with CPython. Also, you can set the environment > variable ``PYPYLOG=jit-summary:-`` and look at the first 2 lines to > see how much time was spent warming up the JIT (or attempting to). > > Note that we did enable PGO long ago, with modest benefits. We gave > up when our JIT compiler became good enough. Maybe now is the time to > try again (and also, PGO itself might have improved in the meantime). > >> We?d like to get some input on how to contribute our optimization recipe to the PyPy dev tree, perhaps by creating an item to the PyPy issue tracker? > > The best would be to create a pull request so that we can look at your > changes more easily. > >> In addition, we would also appreciate any other benchmark or real world use based workload as alternatives to evaluate this. > > You can take any Python program that runs either very shortly or not > faster than CPython. For a larger example (with Python 2.7): > > cd rpython/jit/tl > python ../../bin/rpython -O2 --source targettlr # 24 secs > pypy ../../bin/rpython -O2 --source targettlr # 39 secs > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From matti.picus at gmail.com Fri Nov 11 04:56:24 2016 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 11 Nov 2016 11:56:24 +0200 Subject: [pypy-dev] speed.pypy.org and old versions Message-ID: The landing page for speed.pypy.org, and the pull-downs on the other pages, are clogged with old results from previous versions. It seems the comparison page http://speed.pypy.org/comparison cannot even pull up the database info, and times out, making it useless. A quick fix could be to remove tags for much of the historical data, say for things more than 3 years old, keeping a smattering of significant versions? What you all think? Matti From matti.picus at gmail.com Fri Nov 11 05:46:19 2016 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 11 Nov 2016 12:46:19 +0200 Subject: [pypy-dev] release tarballs of the latest PyPy2.7 available on bitbucket Message-ID: https://bitbucket.org/pypy/pypy/downloads Please download and try them out, if there are no problems we will try for a Sunday release Matti From cfbolz at gmx.de Fri Nov 11 08:34:30 2016 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 11 Nov 2016 14:34:30 +0100 Subject: [pypy-dev] speed.pypy.org and old versions In-Reply-To: References: Message-ID: <60f1f4dc-d085-42a6-b7f4-75579d03da55@email.android.com> Hi Matti, Sounds good to me! Maybe we can keep the tag info around somewhere else (a file in extradoc?) in case we ever need it again. Thanks for doing this! Carl Friedrich On November 11, 2016 10:56:24 AM GMT+01:00, Matti Picus wrote: >The landing page for speed.pypy.org, and the pull-downs on the other >pages, are clogged with old results from previous versions. >It seems the comparison page http://speed.pypy.org/comparison cannot >even pull up the database info, and times out, making it useless. >A quick fix could be to remove tags for much of the historical data, >say >for things more than 3 years old, keeping a smattering of significant >versions? >What you all think? >Matti >_______________________________________________ >pypy-dev mailing list >pypy-dev at python.org >https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.xihong.wang at intel.com Fri Nov 11 15:02:37 2016 From: peter.xihong.wang at intel.com (Wang, Peter Xihong) Date: Fri, 11 Nov 2016 20:02:37 +0000 Subject: [pypy-dev] PGO Optimized Binary Message-ID: <371EBC7881C7844EAAF5556BFF21BCCC42E8B063@ORSMSX105.amr.corp.intel.com> Hi Maciej, This PGO is done for the interpreted code not the JITed code. Yash would give more details on this. Part of reason prompted us to do this work is due to 80% of the CPU cycles being spent in interpreted mode, while running OpenStack Swift in a real world scaled cluster setup. We've got 2% gain from this PGO effort on OpenStack, which is not bad at all. Thanks, Peter ? -----Original Message----- From: pypy-dev [mailto:pypy-dev-bounces+peter.xihong.wang=intel.com at python.org] On Behalf Of pypy-dev-request at python.org Sent: Friday, November 11, 2016 9:00 AM To: pypy-dev at python.org Subject: pypy-dev Digest, Vol 67, Issue 9 Send pypy-dev mailing list submissions to pypy-dev at python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/pypy-dev or, via email, send a message with subject or body 'help' to pypy-dev-request at python.org You can reach the person managing the list at pypy-dev-owner at python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of pypy-dev digest..." Today's Topics: 1. Re: PGO Optimized Binary (Maciej Fijalkowski) 2. speed.pypy.org and old versions (Matti Picus) 3. release tarballs of the latest PyPy2.7 available on bitbucket (Matti Picus) 4. Re: speed.pypy.org and old versions (Carl Friedrich Bolz) ---------------------------------------------------------------------- Message: 1 Date: Thu, 10 Nov 2016 20:42:37 +0200 From: Maciej Fijalkowski To: "Singh, Yashwardhan" Cc: Armin Rigo , "pypy-dev at python.org" Subject: Re: [pypy-dev] PGO Optimized Binary Message-ID: Content-Type: text/plain; charset=UTF-8 Hi 8% of that is very good if you can reproduce it across multiple runs (there is a pretty high variance I would think). You can also try running with --jit off. This gives you an indication of the speed of interpreter, which is a part of warmup On Wed, Nov 9, 2016 at 12:30 AM, Singh, Yashwardhan wrote: > Hi Armin, > > > Thanks for your feedback. > We ran one of the program suggested by you as an example for evaluation: > cd rpython/jit/tl > non-pgo-pypy ../../bin/rpython -O2 --source targettlr pgo-pypy > ../../bin/rpython -O2 --source targettlr > > We got the following results : > Non-Pgo pypy - > [Timer] Timings: > [Timer] annotate --- 7.5 s > [Timer] rtype_lltype --- 5.8 s > [Timer] backendopt_lltype --- 3.6 s > [Timer] stackcheckinsertion_lltype --- 0.1 s > [Timer] database_c --- 19.6 s > [Timer] source_c --- 2.6 s > [Timer] ========================================= > [Timer] Total: --- 39.2 s > > PGO-pypy : > [Timer] Timings: > [Timer] annotate --- 7.6 s > [Timer] rtype_lltype --- 5.1 s > [Timer] backendopt_lltype --- 3.1 s > [Timer] stackcheckinsertion_lltype --- 0.0 s > [Timer] database_c --- 18.5 s > [Timer] source_c --- 2.3 s > [Timer] ========================================= > [Timer] Total: --- 36.6 s > > The delta in performance between these two is about 8%. > > We are working on getting the data to identify the % of interpreted code vs the jited code for both the binaries. We are also working on creating a pull request to get a better feedback on the change. > > Regards > Yash > > ________________________________________ > From: Armin Rigo [armin.rigo at gmail.com] > Sent: Wednesday, November 02, 2016 2:18 AM > To: Singh, Yashwardhan > Cc: pypy-dev at python.org > Subject: Re: [pypy-dev] PGO Optimized Binary > > Hi, > > On 31 October 2016 at 22:28, Singh, Yashwardhan > wrote: >> We applied compiler assisted optimization technique called PGO or Profile Guided Optimization while building PyPy, and found performance got improved by up to 22.4% on the Grand Unified Python Benchmark (GUPB) from ?hg clone https://hg.python.org/benchmarks?. The below result table shows majority of 51 micros got performance boost with 8 got performance regression. > > The kind of performance improvement you are measuring involves only > short- or very short-running programs. A few years ago we'd have > shrugged it off as irrelevant---"please modify the benchmarks so that > they run for at least 10 seconds, more if they are larger"---because > the JIT compiler doesn't have a chance to warm up. But we'd also have > shrugged off your whole attempt---"PGO optimization cannot change > anything to the speed of JIT-produced machine code". > > Nowadays we tend to look more seriously at the cold or warming-up > performance too, or at least we know that we should look there. There > are (stalled) plans of setting up a second benchmark suite for PyPy > which focuses on this. > > You can get an estimate of whether you're looking at cold or hot code: > compare the timings with CPython. Also, you can set the environment > variable ``PYPYLOG=jit-summary:-`` and look at the first 2 lines to > see how much time was spent warming up the JIT (or attempting to). > > Note that we did enable PGO long ago, with modest benefits. We gave > up when our JIT compiler became good enough. Maybe now is the time to > try again (and also, PGO itself might have improved in the meantime). > >> We?d like to get some input on how to contribute our optimization recipe to the PyPy dev tree, perhaps by creating an item to the PyPy issue tracker? > > The best would be to create a pull request so that we can look at your > changes more easily. > >> In addition, we would also appreciate any other benchmark or real world use based workload as alternatives to evaluate this. > > You can take any Python program that runs either very shortly or not > faster than CPython. For a larger example (with Python 2.7): > > cd rpython/jit/tl > python ../../bin/rpython -O2 --source targettlr # 24 secs > pypy ../../bin/rpython -O2 --source targettlr # 39 secs > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev ------------------------------ Message: 2 Date: Fri, 11 Nov 2016 11:56:24 +0200 From: Matti Picus To: "pypy-dev at python.org" Subject: [pypy-dev] speed.pypy.org and old versions Message-ID: Content-Type: text/plain; charset=utf-8; format=flowed The landing page for speed.pypy.org, and the pull-downs on the other pages, are clogged with old results from previous versions. It seems the comparison page http://speed.pypy.org/comparison cannot even pull up the database info, and times out, making it useless. A quick fix could be to remove tags for much of the historical data, say for things more than 3 years old, keeping a smattering of significant versions? What you all think? Matti ------------------------------ Message: 3 Date: Fri, 11 Nov 2016 12:46:19 +0200 From: Matti Picus To: "pypy-dev at python.org" Subject: [pypy-dev] release tarballs of the latest PyPy2.7 available on bitbucket Message-ID: Content-Type: text/plain; charset=utf-8; format=flowed https://bitbucket.org/pypy/pypy/downloads Please download and try them out, if there are no problems we will try for a Sunday release Matti ------------------------------ Message: 4 Date: Fri, 11 Nov 2016 14:34:30 +0100 From: Carl Friedrich Bolz To: Matti Picus , "pypy-dev at python.org" Subject: Re: [pypy-dev] speed.pypy.org and old versions Message-ID: <60f1f4dc-d085-42a6-b7f4-75579d03da55 at email.android.com> Content-Type: text/plain; charset="utf-8" Hi Matti, Sounds good to me! Maybe we can keep the tag info around somewhere else (a file in extradoc?) in case we ever need it again. Thanks for doing this! Carl Friedrich On November 11, 2016 10:56:24 AM GMT+01:00, Matti Picus wrote: >The landing page for speed.pypy.org, and the pull-downs on the other >pages, are clogged with old results from previous versions. >It seems the comparison page http://speed.pypy.org/comparison cannot >even pull up the database info, and times out, making it useless. >A quick fix could be to remove tags for much of the historical data, >say for things more than 3 years old, keeping a smattering of >significant versions? >What you all think? >Matti >_______________________________________________ >pypy-dev mailing list >pypy-dev at python.org >https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Subject: Digest Footer _______________________________________________ pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev ------------------------------ End of pypy-dev Digest, Vol 67, Issue 9 *************************************** From matti.picus at gmail.com Sat Nov 12 11:40:05 2016 From: matti.picus at gmail.com (Matti Picus) Date: Sat, 12 Nov 2016 18:40:05 +0200 Subject: [pypy-dev] PyPy2.7-v5.6.0 has been released Message-ID: <892693b7-6c57-c83b-3d41-6ea19c72926a@gmail.com> PyPy2.7 v5.6 released - stdlib 2.7.12 support, C-API improvements, and more https://morepypy.blogspot.com/2016/11/pypy27-v56-released-stdlib-2712-support.html Thanks to all involved, please help get the word out. Success stories would be nice, we can help you do a guest blog post or link to your own posts. Matti From drtomc at gmail.com Fri Nov 25 21:32:16 2016 From: drtomc at gmail.com (Thomas Conway) Date: Sat, 26 Nov 2016 13:32:16 +1100 Subject: [pypy-dev] array sorting Message-ID: Hi Pypies, I'm new to the list, so let me start by saying I totally love pypy and am using it for my bioinformatics codes. I realise this isn't exactly a pypy specific problem, by the standard array module doesn't provide sort() on arrays. In consequence you either have to convert them to lists and use list.sort() or you have to implement sort on them yourself. I've implemented several sorts (radix, american flag, quick sort, in-place heap sort) and they all perform slower than converting to a list and sorting that. It strikes me that any python implementation should benefit from an array.sort() method since unboxing/type checking is unnecessary. What is the wisdom of this community on the matter? It's something I'd consider taking on if it was generally deemed worthwhile. Tom. -- Thomas Conway drtomc at gmail.com My friends, love is better than anger. Hope is better than fear. Optimism is better than despair. So let us be loving, hopeful and optimistic. And we'll change the world. - Jack Layton -------------- next part -------------- An HTML attachment was scrubbed... URL: From shubharamani at yahoo.com Sat Nov 26 11:03:26 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Sat, 26 Nov 2016 16:03:26 +0000 (UTC) Subject: [pypy-dev] Can't run any puppy tests, get --assert: invalid choice etc... References: <1515684085.665022.1480176206705.ref@mail.yahoo.com> Message-ID: <1515684085.665022.1480176206705@mail.yahoo.com> For instance this command :usage: py.test [options] [file_or_dir] [file_or_dir] [...]py.test: error: argument --assert: invalid choice: 'reinterp' (choose from 'rewrite', 'plain')?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From shubharamani at yahoo.com Sat Nov 26 11:04:45 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Sat, 26 Nov 2016 16:04:45 +0000 (UTC) Subject: [pypy-dev] Can't run any puppy tests, get --assert: invalid choice etc... In-Reply-To: <1515684085.665022.1480176206705@mail.yahoo.com> References: <1515684085.665022.1480176206705.ref@mail.yahoo.com> <1515684085.665022.1480176206705@mail.yahoo.com> Message-ID: <518346201.677665.1480176285979@mail.yahoo.com> pypy not puppy (spell-check woes)?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com On Saturday, November 26, 2016 8:03 AM, Shubha Ramani via pypy-dev wrote: For instance this command :usage: py.test [options] [file_or_dir] [file_or_dir] [...]py.test: error: argument --assert: invalid choice: 'reinterp' (choose from 'rewrite', 'plain')?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com _______________________________________________ pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From shubharamani at yahoo.com Sat Nov 26 11:58:02 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Sat, 26 Nov 2016 16:58:02 +0000 (UTC) Subject: [pypy-dev] Can't run any puppy tests, get --assert: invalid choice etc... In-Reply-To: <518346201.677665.1480176285979@mail.yahoo.com> References: <1515684085.665022.1480176206705.ref@mail.yahoo.com> <1515684085.665022.1480176206705@mail.yahoo.com> <518346201.677665.1480176285979@mail.yahoo.com> Message-ID: <1013261438.694193.1480179482504@mail.yahoo.com> Easily solved. Do the following:1)in?./pytest.ini and?./rpython/pytest.ini?change assert=reinterp to assert=rewrite. It turns out in the latest version of pytest assert=reinterp is no longer supported2)comment out the following lines from?~/pypy/pypy/tool/pytest/appsupport.py #try:#? ? from _pytest.assertion.reinterpret import reinterpret as interpret#except ImportError:#? ? from _pytest.assertion.newinterpret import interpret # _____?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com On Saturday, November 26, 2016 8:04 AM, Shubha Ramani wrote: pypy not puppy (spell-check woes)?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com On Saturday, November 26, 2016 8:03 AM, Shubha Ramani via pypy-dev wrote: For instance this command :usage: py.test [options] [file_or_dir] [file_or_dir] [...]py.test: error: argument --assert: invalid choice: 'reinterp' (choose from 'rewrite', 'plain')?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com _______________________________________________ pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostia.lopuhin at gmail.com Sat Nov 26 15:35:19 2016 From: kostia.lopuhin at gmail.com (Konstantin Lopuhin) Date: Sat, 26 Nov 2016 23:35:19 +0300 Subject: [pypy-dev] array sorting In-Reply-To: References: Message-ID: Hello Thomas, I think a library with sort on arrays can useful, for example I needed radix sort in one project and would have used a library if such library existed :) 2016-11-26 5:32 GMT+03:00 Thomas Conway : > Hi Pypies, > > I'm new to the list, so let me start by saying I totally love pypy and am > using it for my bioinformatics codes. > > I realise this isn't exactly a pypy specific problem, by the standard array > module doesn't provide sort() on arrays. In consequence you either have to > convert them to lists and use list.sort() or you have to implement sort on > them yourself. I've implemented several sorts (radix, american flag, quick > sort, in-place heap sort) and they all perform slower than converting to a > list and sorting that. > > It strikes me that any python implementation should benefit from an > array.sort() method since unboxing/type checking is unnecessary. > > What is the wisdom of this community on the matter? It's something I'd > consider taking on if it was generally deemed worthwhile. > > Tom. > -- > Thomas Conway > drtomc at gmail.com > My friends, love is better than anger. Hope is better than fear. Optimism is > better than despair. So let us be loving, hopeful and optimistic. And we'll > change the world. - Jack Layton > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From armin.rigo at gmail.com Sun Nov 27 03:44:27 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Sun, 27 Nov 2016 09:44:27 +0100 Subject: [pypy-dev] Can't run any puppy tests, get --assert: invalid choice etc... In-Reply-To: <1013261438.694193.1480179482504@mail.yahoo.com> References: <1515684085.665022.1480176206705.ref@mail.yahoo.com> <1515684085.665022.1480176206705@mail.yahoo.com> <518346201.677665.1480176285979@mail.yahoo.com> <1013261438.694193.1480179482504@mail.yahoo.com> Message-ID: Hi Shubha, On 26 November 2016 at 17:58, Shubha Ramani via pypy-dev wrote: > change assert=reinterp to assert=rewrite. It turns out in the latest version > of pytest assert=reinterp is no longer supported It means that the latest version of py.test can't be used, then. With your hack you disable a feature that is essential to our development. Note that PyPy could, so far, use an external py.test, but always also came with its own included version, available as the top-level "pytest.py". Maybe we should systematically document using that one instead and make sure it doesn't pick any globally installed "_pytest" package. There was also another essential-to-PyPy feature that was removed recently, so maybe we'll stop updating py.test altogether in the future. A bient?t, Armin. From armin.rigo at gmail.com Sun Nov 27 04:36:15 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Sun, 27 Nov 2016 10:36:15 +0100 Subject: [pypy-dev] array sorting In-Reply-To: References: Message-ID: Hi Thomas, On 26 November 2016 at 03:32, Thomas Conway wrote: > I realise this isn't exactly a pypy specific problem, by the standard array > module doesn't provide sort() on arrays. I fear it's not PyPy's place to add such new methods to standard Python modules. You'd need to go to CPython and discuss the issue with them---which is not really a fun process imho. However, here are a few paths: * First, do you really need the array module? If you use a list containing only signed, machine-sized integers, then that list is represented in PyPy as compactly as an array of the 'l' type. It won't work if you really need more compact arrays, but if you don't, then one solution to your problem is simply to use lists and not arrays in the first place. * Or, use cffi to call qsort(). This is simpler (and most probably faster) than writing the sort in pure Python. It works with an array object from the array module, or anything supporting the buffer interface: ``lib.qsort(ffi.from_buffer(my_array), more_args...)``. You'd need to use the ``set_source()`` mode of CFFI in order to write an efficient comparison function in C to pass as argument. * (Additionally, you can use cffi's ``ffi.new("int[]", length)`` in the first place instead of the array module, as it is similar but more flexible: see http://cffi.readthedocs.io/en/latest/overview.html#struct-array-example-minimal-in-line . This does not really change anything though.) * The above seem like more general solutions than adding a sort method to the array module. However, if you really want to do that, then maybe create a function ``__pypy__.sort_buffer()`` instead, able to sort a buffer of integers of some size; it would not be on the array module specifically, and clearly it would be a pypy extension. A bient?t, Armin. From shubharamani at yahoo.com Sun Nov 27 07:44:35 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Sun, 27 Nov 2016 04:44:35 -0800 Subject: [pypy-dev] Can't run any puppy tests, get --assert: invalid choice etc... In-Reply-To: References: <1515684085.665022.1480176206705.ref@mail.yahoo.com> <1515684085.665022.1480176206705@mail.yahoo.com> <518346201.677665.1480176285979@mail.yahoo.com> <1013261438.694193.1480179482504@mail.yahoo.com> Message-ID: Wow. Thanks for the tip Armin. Shubha > On Nov 27, 2016, at 12:44 AM, Armin Rigo wrote: > > Hi Shubha, > > On 26 November 2016 at 17:58, Shubha Ramani via pypy-dev > wrote: >> change assert=reinterp to assert=rewrite. It turns out in the latest version >> of pytest assert=reinterp is no longer supported > > It means that the latest version of py.test can't be used, then. With > your hack you disable a feature that is essential to our development. > > Note that PyPy could, so far, use an external py.test, but always also > came with its own included version, available as the top-level > "pytest.py". Maybe we should systematically document using that one > instead and make sure it doesn't pick any globally installed "_pytest" > package. > > There was also another essential-to-PyPy feature that was removed > recently, so maybe we'll stop updating py.test altogether in the > future. > > > A bient?t, > > Armin. From shubharamani at yahoo.com Mon Nov 28 15:24:46 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Mon, 28 Nov 2016 20:24:46 +0000 (UTC) Subject: [pypy-dev] Sample benchmarks to run through pypy JIT ? References: <35826639.1761374.1480364686298.ref@mail.yahoo.com> Message-ID: <35826639.1761374.1480364686298@mail.yahoo.com> Hello. I'm taking over development of the vtune branch. I do work for Intelbut it's easier to use my personal account. Question, as I develop this branch can someone recommend some good?Python benchmarks out there which have been tested with PyPy and?have seen performance improvements ? pypy / pypy / vtune ? Bitbucket | | | | | | | | | | | pypy / pypy / vtune ? Bitbucket | | | | ?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From shubharamani at yahoo.com Mon Nov 28 16:51:17 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Mon, 28 Nov 2016 21:51:17 +0000 (UTC) Subject: [pypy-dev] Sample benchmarks to run through pypy JIT ? In-Reply-To: <35826639.1761374.1480364686298@mail.yahoo.com> References: <35826639.1761374.1480364686298.ref@mail.yahoo.com> <35826639.1761374.1480364686298@mail.yahoo.com> Message-ID: <937662812.1823982.1480369877866@mail.yahoo.com> I found this website which has a lot of good benchmarks:pypy / benchmarks / Downloads ? Bitbucket | | | | | | | | | | | pypy / benchmarks / Downloads ? Bitbucket | | | | ?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com On Monday, November 28, 2016 12:24 PM, Shubha Ramani wrote: Hello. I'm taking over development of the vtune branch. I do work for Intelbut it's easier to use my personal account. Question, as I develop this branch can someone recommend some good?Python benchmarks out there which have been tested with PyPy and?have seen performance improvements ? pypy / pypy / vtune ? Bitbucket | | | | | | | | | | | pypy / pypy / vtune ? Bitbucket | | | | ?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Tue Nov 29 05:03:43 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Tue, 29 Nov 2016 11:03:43 +0100 Subject: [pypy-dev] Sample benchmarks to run through pypy JIT ? In-Reply-To: <35826639.1761374.1480364686298@mail.yahoo.com> References: <35826639.1761374.1480364686298.ref@mail.yahoo.com> <35826639.1761374.1480364686298@mail.yahoo.com> Message-ID: Hi, On 28 November 2016 at 21:24, Shubha Ramani via pypy-dev wrote: > Question, as I develop this branch can someone recommend some good > Python benchmarks out there which have been tested with PyPy and > have seen performance improvements ? http://speed.pypy.org/ Armin From shubharamani at yahoo.com Tue Nov 29 17:41:04 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Tue, 29 Nov 2016 22:41:04 +0000 (UTC) Subject: [pypy-dev] assert error assert sys.maxint == (2**31 - 1) References: <2073195295.2666278.1480459264971.ref@mail.yahoo.com> Message-ID: <2073195295.2666278.1480459264971@mail.yahoo.com> Why ? When I run rpython:/opt/pypy/rpython$ ./bin/rpython -Ojit ?--jit-backend=x86 ./translator/goal/targettestdicts.py? [translation:info] Error:? ?File "/opt/pypy/rpython/translator/goal/translate.py", line 318, in main? ? drv.proceed(goals)? ?File "/opt/pypy/rpython/translator/driver.py", line 551, in proceed? ? result = self._execute(goals, task_skip = self._maybe_skip())? ?File "/opt/pypy/rpython/translator/tool/taskengine.py", line 114, in _execute? ? res = self._do(goal, taskcallable, *args, **kwds)? ?File "/opt/pypy/rpython/translator/driver.py", line 278, in _do? ? res = func()? ?File "/opt/pypy/rpython/translator/driver.py", line 361, in task_pyjitpl_lltype? ? backend_name=self.config.translation.jit_backend, inline=True)? ?File "/opt/pypy/rpython/jit/metainterp/warmspot.py", line 49, in apply_jit? ? **kwds)? ?File "/opt/pypy/rpython/jit/metainterp/warmspot.py", line 224, in __init__? ? self.build_cpu(CPUClass, **kwds)? ?File "/opt/pypy/rpython/jit/metainterp/warmspot.py", line 473, in build_cpu? ? translate_support_code, gcdescr=self.gcdescr)? ?File "/opt/pypy/rpython/jit/backend/x86/runner.py", line 143, in __init__? ? assert sys.maxint == (2**31 - 1)[translation:ERROR] AssertionError[translation] start debugger...> /opt/pypy/rpython/jit/backend/x86/runner.py(143)__init__()-> assert sys.maxint == (2**31 - 1)(Pdb+)? ?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From shubharamani at yahoo.com Tue Nov 29 17:52:49 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Tue, 29 Nov 2016 22:52:49 +0000 (UTC) Subject: [pypy-dev] assert error assert sys.maxint == (2**31 - 1) In-Reply-To: <2073195295.2666278.1480459264971@mail.yahoo.com> References: <2073195295.2666278.1480459264971.ref@mail.yahoo.com> <2073195295.2666278.1480459264971@mail.yahoo.com> Message-ID: <516357598.2662965.1480459969028@mail.yahoo.com> When I change backend to auto:./bin/rpython -Ojit ?--jit-backend=auto --cc=gcc ?./translator/goal/targetpushpop.py I get...translation:ERROR] Exception: no jit_merge_point found![translation] start debugger...> /opt/pypy/rpython/jit/metainterp/warmspot.py(169)find_jit_merge_points()-> raise Exception("no jit_merge_point found!")(Pdb+)? ?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com On Tuesday, November 29, 2016 2:41 PM, Shubha Ramani wrote: Why ? When I run rpython:/opt/pypy/rpython$ ./bin/rpython -Ojit ?--jit-backend=x86 ./translator/goal/targettestdicts.py? [translation:info] Error:? ?File "/opt/pypy/rpython/translator/goal/translate.py", line 318, in main? ? drv.proceed(goals)? ?File "/opt/pypy/rpython/translator/driver.py", line 551, in proceed? ? result = self._execute(goals, task_skip = self._maybe_skip())? ?File "/opt/pypy/rpython/translator/tool/taskengine.py", line 114, in _execute? ? res = self._do(goal, taskcallable, *args, **kwds)? ?File "/opt/pypy/rpython/translator/driver.py", line 278, in _do? ? res = func()? ?File "/opt/pypy/rpython/translator/driver.py", line 361, in task_pyjitpl_lltype? ? backend_name=self.config.translation.jit_backend, inline=True)? ?File "/opt/pypy/rpython/jit/metainterp/warmspot.py", line 49, in apply_jit? ? **kwds)? ?File "/opt/pypy/rpython/jit/metainterp/warmspot.py", line 224, in __init__? ? self.build_cpu(CPUClass, **kwds)? ?File "/opt/pypy/rpython/jit/metainterp/warmspot.py", line 473, in build_cpu? ? translate_support_code, gcdescr=self.gcdescr)? ?File "/opt/pypy/rpython/jit/backend/x86/runner.py", line 143, in __init__? ? assert sys.maxint == (2**31 - 1)[translation:ERROR] AssertionError[translation] start debugger...> /opt/pypy/rpython/jit/backend/x86/runner.py(143)__init__()-> assert sys.maxint == (2**31 - 1)(Pdb+)? ?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Tue Nov 29 19:05:56 2016 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Tue, 29 Nov 2016 16:05:56 -0800 (PST) Subject: [pypy-dev] assert error assert sys.maxint == (2**31 - 1) In-Reply-To: <516357598.2662965.1480459969028@mail.yahoo.com> References: <2073195295.2666278.1480459264971.ref@mail.yahoo.com> <2073195295.2666278.1480459264971@mail.yahoo.com> <516357598.2662965.1480459969028@mail.yahoo.com> Message-ID: Shubha, On Tuesday 2016-11-29 22:52, Shubha Ramani via pypy-dev wrote: > When I change backend to auto:./bin/rpython -Ojit --jit-backend=auto --cc=gcc ./translator/goal/targetpushpop.py those small translator exampled do not use a jit driver and thus have no jit_merge_point. You can add one: from rpython.rlib import jit driver = jit.JitDriver(greens = [], reds = 'auto') and then for example at the opening of entry_point(): driver.jit_merge_point() See here for details: http://rpython.readthedocs.io/en/latest/jit/pyjitpl5.html > When I run rpython:/opt/pypy/rpython$ ./bin/rpython -Ojit --jit-backend=x86 ./translator/goal/targettestdicts.py Is probably b/c you want the x86_64 backend. That seems to be missing from the set of options in config/translationoption.py+121, but 'auto' (default) will select it for your machine. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From shubharamani at yahoo.com Tue Nov 29 19:21:57 2016 From: shubharamani at yahoo.com (Shubha Ramani) Date: Wed, 30 Nov 2016 00:21:57 +0000 (UTC) Subject: [pypy-dev] assert error assert sys.maxint == (2**31 - 1) In-Reply-To: References: <2073195295.2666278.1480459264971.ref@mail.yahoo.com> <2073195295.2666278.1480459264971@mail.yahoo.com> <516357598.2662965.1480459969028@mail.yahoo.com> Message-ID: <1171963270.2724885.1480465317495@mail.yahoo.com> thanks WLavrijsen, your tips worked. Shubha?Shubha D. Ramanishubharamani at gmail.com shubharamani at yahoo.com On Tuesday, November 29, 2016 4:06 PM, "wlavrijsen at lbl.gov" wrote: Shubha, On Tuesday 2016-11-29 22:52, Shubha Ramani via pypy-dev wrote: > When I change backend to auto:./bin/rpython -Ojit --jit-backend=auto --cc=gcc ./translator/goal/targetpushpop.py those small translator exampled do not use a jit driver and thus have no jit_merge_point. You can add one: ? from rpython.rlib import jit ? driver = jit.JitDriver(greens = [], reds = 'auto') and then for example at the opening of entry_point(): ? driver.jit_merge_point() See here for details: ? http://rpython.readthedocs.io/en/latest/jit/pyjitpl5.html > When I run rpython:/opt/pypy/rpython$ ./bin/rpython -Ojit --jit-backend=x86 ./translator/goal/targettestdicts.py Is probably b/c you want the x86_64 backend. That seems to be missing from the set of options in config/translationoption.py+121, but 'auto' (default) will select it for your machine. Best regards, ? ? ? ? ? ? Wim -- WLavrijsen at lbl.gov? ? --? ? +1 (510) 486 6411? ? --? ? www.lavrijsen.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Wed Nov 30 05:57:42 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 30 Nov 2016 11:57:42 +0100 Subject: [pypy-dev] Bump DARWIN_VERSION_MIN to 10.7? In-Reply-To: References: Message-ID: Hi all OS/X users, On 8 October 2016 at 12:59, Jeremy Thurgood wrote: > I recently tried to build RevDB on OS X and the build failed because > __thread isn't supported on OS X 10.6 which RPython specifies as the > minimum supported version. > > I've updated DARWIN_VERSION_MIN to 10.7 in the reverse-debugger > branch[1], because otherwise it doesn't build at all on OS X. Armin > suggested that there would be a performance benefit to increasing > DARWIN_VERSION_MIN for pypy as well (along with adding darwin to the > SUPPORT__THREAD list). I would like to make this change, but I don't > want to break pypy for anyone who may still need OS X 10.6 support. Someone with OS/ user should look at this as well: https://bitbucket.org/pypy/pypy/issues/2432/problems-with-pypy-on-osx-dyld-symbol-not There are maybe other similar places, too; you'd need to grep around to find them. Making all such places dependent on a single variable, like this DARWIN_VERSION_MIN, would be a good start. Then we can decide to bump the minimum OS/X version to 10.7, to get __thread in the normal PyPy too. This involves making sure that __thread is really used systematically then, i.e. there is no "if platform == darwin" that would disable their use here and there. If no-one shows up, the current situation will not change (not using __thread has performance implications, notably in calls to C functions around which we release the GIL). A bient?t, Armin. From John.Zhang at anu.edu.au Tue Nov 29 22:14:44 2016 From: John.Zhang at anu.edu.au (John Zhang) Date: Wed, 30 Nov 2016 03:14:44 +0000 Subject: [pypy-dev] RPython Darwin platform description so_prefixes empty Message-ID: <5053B1B0-2CF6-445F-83D2-005B14E26A92@anu.edu.au> Hi all, I have been doing some coding using RFFI to load function pointers from shared libraries (`llexternal` function). However what bothers me is that apparently `Darwin_x86_64` platform class (defined in translator/platform/darwin.py) does not include ?lib? in `so_prefixes`. This is causing me some problems with not being able to find my shared libraries on macOS X. I?m wondering if that?s intentional, or just a mistake? It should be a simple fix if it?s a mistake. Regards, John Zhang ------------------------------------------------------ John Zhang Research Assistant Programming Languages, Design & Implementation Division Computer Systems Group ANU College of Engineering & Computer Science 108 North Rd The Australian National University Acton ACT 2601 john.zhang at anu.edu.au -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Wed Nov 30 12:56:46 2016 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 30 Nov 2016 18:56:46 +0100 Subject: [pypy-dev] RPython Darwin platform description so_prefixes empty In-Reply-To: <5053B1B0-2CF6-445F-83D2-005B14E26A92@anu.edu.au> References: <5053B1B0-2CF6-445F-83D2-005B14E26A92@anu.edu.au> Message-ID: Hi John, On 30 November 2016 at 04:14, John Zhang wrote: > `so_prefixes`. This is causing me some problems with not being able to find > my shared libraries on macOS X. > I?m wondering if that?s intentional, or just a mistake? It should be a > simple fix if it?s a mistake. I don't know, and that's a good question. But the first question is: how does it work for PyPy on OS/X? All (or most) tests pass there. Can you pinpoint the difference? If not, can you point us to your version of the code, so that we can have a guess at what the difference might be? A bient?t, Armin.