From matti.picus at gmail.com Tue Mar 9 07:53:19 2021 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 9 Mar 2021 14:53:19 +0200 Subject: [pypy-dev] moving blog posts to pypy.org In-Reply-To: <63595332-b0f8-7b8b-5f3e-6b72750d69fa@gmail.com> References: <009345fb-f871-b2ff-5a48-8322dd9b91f1@gmail.com> <63595332-b0f8-7b8b-5f3e-6b72750d69fa@gmail.com> Message-ID: On 2/28/21 10:51 AM, Matti Picus wrote: > > I opened a repo on https://github.com/pypy/pypy.org. It now completes > the points at the end of my mail: preview for PRs via netlify, uses > utterances for new blog posts. Please take a look at the deploy > preview https://keen-mestorf-442210.netlify.app, especially the new > Blog dropdown and links. > > > Matti > The move is ready. I have - made a PR to the python salt recipe to pull the static site from the new repo https://github.com/python/psf-salt/pull/205 - created a blog post for morepypy.blogspot.com, here is the preview https://www.blogger.com/blog/post/edit/preview/3971202189709462152/2449361080661885989 - made a parallel PR to the new repo with the same content, here is the PR https://github.com/pypy/pypy.org/pull/7 and the preview (comments should work when it goes live) https://deploy-preview-7--keen-mestorf-442210.netlify.app/posts/2021/03/pypys-blog-has-moved.html Last chance to stop the move is now. Comments or fixes/typos to the blog post(s) are also welcome. Matti From hodgestar at gmail.com Tue Mar 9 08:40:25 2021 From: hodgestar at gmail.com (Simon Cross) Date: Tue, 9 Mar 2021 15:40:25 +0200 Subject: [pypy-dev] moving blog posts to pypy.org In-Reply-To: References: <009345fb-f871-b2ff-5a48-8322dd9b91f1@gmail.com> <63595332-b0f8-7b8b-5f3e-6b72750d69fa@gmail.com> Message-ID: Thanks Matti. The post looks good to me. From armin.rigo at gmail.com Wed Mar 10 00:13:15 2021 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 10 Mar 2021 06:13:15 +0100 Subject: [pypy-dev] moving blog posts to pypy.org In-Reply-To: References: <009345fb-f871-b2ff-5a48-8322dd9b91f1@gmail.com> <63595332-b0f8-7b8b-5f3e-6b72750d69fa@gmail.com> Message-ID: Hi Matti, On Tue, 9 Mar 2021 at 13:53, Matti Picus wrote: > Last chance to stop the move is now. Comments or fixes/typos to the blog > post(s) are also welcome. Thank you for your continued work! Armin From matti.picus at gmail.com Mon Mar 15 08:04:29 2021 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 15 Mar 2021 14:04:29 +0200 Subject: [pypy-dev] Release 7.3.4 Message-ID: <770265fb-61cb-a3f5-bbca-ed9a942dbf85@gmail.com> I would like to release new versions of PyPy for python2.7 and python3.7. I think 3.7 has matured to the point that we can drop both the "beta" label and the python3.6 release. The python3.8 version is not yet ready for an "alpha" label. There are three issues (one about running "pypy -Wonce ..." and warnings inside a decorator, and two having to do with sqlite3) marked as blockers [0]. Are there more blockers? Anyone want to take a look at fixing them? Matti [0] https://foss.heptapod.net/pypy/pypy/-/milestones/3 From strombrg at gmail.com Mon Mar 15 18:16:28 2021 From: strombrg at gmail.com (Dan Stromberg) Date: Mon, 15 Mar 2021 15:16:28 -0700 Subject: [pypy-dev] Program slower on Pypy 7.3.3 (3.7.9) than CPython 3.9. Message-ID: And it's opensource, though many of the inputs are licensed. The code is at https://stromberg.dnsalias.org/~strombrg/music-pipeline/ ( https://stromberg.dnsalias.org/svn/music-pipeline/trunk/) It appears to be more than 10x slower. I haven't profiled it yet. I believe it's probably the "Blocklisting files..." part that's slow. That part is O(n*m), and seems to take forever. It's heavy on regular expressions. Are regular expressions expected to be slow on Pypy3? Thanks. -- Dan Stromberg -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Tue Mar 16 00:20:14 2021 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 15 Mar 2021 21:20:14 -0700 Subject: [pypy-dev] Program slower on Pypy 7.3.3 (3.7.9) than CPython 3.9. In-Reply-To: References: Message-ID: I put a small SSCCE for this at https://stromberg.dnsalias.org/svn/regex-fodder/trunk It's almost 10x slower, not quite as much as music-pipeline. On Mon, Mar 15, 2021 at 3:17 PM Dan Stromberg wrote: > > And it's opensource, though many of the inputs are licensed. > > The code is at https://stromberg.dnsalias.org/~strombrg/music-pipeline/ ( > https://stromberg.dnsalias.org/svn/music-pipeline/trunk/) > > It appears to be more than 10x slower. > > I haven't profiled it yet. I believe it's probably the "Blocklisting > files..." part that's slow. That part is O(n*m), and seems to take > forever. It's heavy on regular expressions. > > Are regular expressions expected to be slow on Pypy3? > > Thanks. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfbolz at gmx.de Tue Mar 16 05:27:20 2021 From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick) Date: Tue, 16 Mar 2021 10:27:20 +0100 Subject: [pypy-dev] Program slower on Pypy 7.3.3 (3.7.9) than CPython 3.9. In-Reply-To: References: Message-ID: <2b661159-f188-8d42-f1f4-43d0f4dd6762@gmx.de> On 3/15/21 11:16 PM, Dan Stromberg wrote: > > And it's opensource, though many of the inputs are licensed. > > The code is at https://stromberg.dnsalias.org/~strombrg/music-pipeline/ > > (https://stromberg.dnsalias.org/svn/music-pipeline/trunk/ > ) > > It appears to be more than 10x slower. > > I haven't profiled it yet.? I believe it's probably the "Blocklisting > files..." part that's slow.? That part is O(n*m), and seems to take > forever.? It's heavy on regular expressions. > > Are regular expressions expected to be slow on Pypy3? Hi Dan, Interesting problem! single regular expressions are reasonably fast on PyPy, being jitted. But I don't think we looked into the problem of "what if you have thousands of them" before. Your reproducer is hitting a kind of known, hard to fix corner case of the JIT, it's actually producing a linear search over the existing regular expressions for every match call in this case, with catastrophic consequences. It's on my mid-term plans to work on this problem, but not next week. Here's a fun workaround, that improves the performance of both CPython (by about 2x for me) and pypy (by 10x or so): turn the many regular expressions into a single one: regex_strings = [f"(?:{one_regex()})" for repno in range(2_046)] regex_compiled = re.compile("|".join(regex_strings)) then you replace the match calls with a single one: for filename in filenames: if regex_compiled.match(filename): matches += 1 I believe you can try the same approach for your full program? Cheers, Carl Friedrich From strombrg at gmail.com Wed Mar 17 13:56:55 2021 From: strombrg at gmail.com (Dan Stromberg) Date: Wed, 17 Mar 2021 10:56:55 -0700 Subject: [pypy-dev] Program slower on Pypy 7.3.3 (3.7.9) than CPython 3.9. In-Reply-To: <2b661159-f188-8d42-f1f4-43d0f4dd6762@gmx.de> References: <2b661159-f188-8d42-f1f4-43d0f4dd6762@gmx.de> Message-ID: On Tue, Mar 16, 2021 at 2:27 AM Carl Friedrich Bolz-Tereick wrote: > On 3/15/21 11:16 PM, Dan Stromberg wrote: > > > > And it's opensource, though many of the inputs are licensed. > > > > The code is at https://stromberg.dnsalias.org/~strombrg/music-pipeline/ > > > > (https://stromberg.dnsalias.org/svn/music-pipeline/trunk/ > > ) > > > > It appears to be more than 10x slower. > > > > I haven't profiled it yet. I believe it's probably the "Blocklisting > > files..." part that's slow. That part is O(n*m), and seems to take > > forever. It's heavy on regular expressions. > > > > Are regular expressions expected to be slow on Pypy3? > > Hi Dan, > > Interesting problem! single regular expressions are reasonably fast on > PyPy, being jitted. But I don't think we looked into the problem of > "what if you have thousands of them" before. Your reproducer is hitting > a kind of known, hard to fix corner case of the JIT, it's actually > producing a linear search over the existing regular expressions for > every match call in this case, with catastrophic consequences. It's on > my mid-term plans to work on this problem, but not next week. > Here's another SSCCE that surprised me a little. I create and del the compiled regexes one at a time, but it's still slow: https://stromberg.dnsalias.org/svn/regex-fodder/trunk/regex-fodder-3 > Here's a fun workaround, that improves the performance of both CPython > (by about 2x for me) and pypy (by 10x or so): turn the many regular > expressions into a single one: > > regex_strings = [f"(?:{one_regex()})" for repno in range(2_046)] > regex_compiled = re.compile("|".join(regex_strings)) > > then you replace the match calls with a single one: > > for filename in filenames: > if regex_compiled.match(filename): > matches += 1 > > I believe you can try the same approach for your full program? > I'm familiar with the technique, as well as that of creating a single, big trie regex. For this application though, I need to check at the end if each regex was matched exactly once, to deter typos causing things to get missed. Thanks much for the suggestion and more! -- Dan Stromberg -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Thu Mar 18 13:32:19 2021 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 18 Mar 2021 10:32:19 -0700 Subject: [pypy-dev] Program slower on Pypy 7.3.3 (3.7.9) than CPython 3.9. In-Reply-To: References: <2b661159-f188-8d42-f1f4-43d0f4dd6762@gmx.de> Message-ID: Hi folks. I've modified my code to use str.startswith instead of re.match. I had a one-to-one correspondence between filenames and regexes anyway, so it doesn't really sacrifice anything. This way the original app (music-pipeline) is nice and fast now on pypy3 7.3.3. I'm leaving the various SSCCE's at https://stromberg.dnsalias.org/svn/regex-fodder/trunk in case someone wants to use them to replicate the problem going forward. They're commented to describe what they do and whether they are fast or slow. As Carl said, the issue seems to be that pypy3 7.3.3 doesn't like having very many regular expressions in the same program - even if only one compiled regex exists at any given time (no-longer-needed regexes disposed of with del). Thanks again! -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfbolz at gmx.de Thu Mar 18 13:38:37 2021 From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick) Date: Thu, 18 Mar 2021 18:38:37 +0100 Subject: [pypy-dev] Program slower on Pypy 7.3.3 (3.7.9) than CPython 3.9. In-Reply-To: References: <2b661159-f188-8d42-f1f4-43d0f4dd6762@gmx.de> Message-ID: Thanks Dan! It's definitely a real problem that you identified, I've filed a bug about it here: https://foss.heptapod.net/pypy/pypy/-/issues/3419 Thanks for the report! Carl Friedrich On March 18, 2021 6:32:19 PM GMT+01:00, Dan Stromberg wrote: >Hi folks. > >I've modified my code to use str.startswith instead of re.match. I had >a >one-to-one correspondence between filenames and regexes anyway, so it >doesn't really sacrifice anything. > >This way the original app (music-pipeline) is nice and fast now on >pypy3 >7.3.3. > >I'm leaving the various SSCCE's at >https://stromberg.dnsalias.org/svn/regex-fodder/trunk in case someone >wants >to use them to replicate the problem going forward. They're commented >to >describe what they do and whether they are fast or slow. As Carl said, >the >issue seems to be that pypy3 7.3.3 doesn't like having very many >regular >expressions in the same program - even if only one compiled regex >exists at >any given time (no-longer-needed regexes disposed of with del). > >Thanks again! -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Fri Mar 19 10:31:33 2021 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 19 Mar 2021 16:31:33 +0200 Subject: [pypy-dev] PyPy 7.3.4rc1 tarballs are up Message-ID: The first release candidate for PyPy7.3.4 is up at https://downloads.python.org/pypy/. People using cibuildwheel or github actions to test PyPy should be able to use this in their test runs. The sha256 checksums can be found in the PR to the pypy.org web site at https://github.com/pypy/pypy.org/pull/13/files#diff-622ff636d3511de2e408d8b4134b4b825743385cb6c8d9bbadf6dc5b0c887f0eR491 or via the preview at https://deploy-preview-13--keen-mestorf-442210.netlify.app/download_advanced.html#checksums The draft release note is up at https://doc.pypy.org/en/latest/release-v7.3.4.html This is an incrementally better release: there are no ground-breaking changes. However there are a number of JIT preformance enhancements, and overall better compatibility with CPython. Please try it out and report issues on IRC or on the issue tracker. Matti From pierre.augier at univ-grenoble-alpes.fr Fri Mar 19 15:27:16 2021 From: pierre.augier at univ-grenoble-alpes.fr (PIERRE AUGIER) Date: Fri, 19 Mar 2021 20:27:16 +0100 (CET) Subject: [pypy-dev] PyPy 7.3.4rc1 tarballs are up In-Reply-To: References: Message-ID: <1991061799.4703128.1616182036810.JavaMail.zimbra@univ-grenoble-alpes.fr> Thank a lot Matti and all PyPy developers for the work done on this release, Two small remarks on the sentence in the release note: "There are also some significant performance improvements around maps (dictionaries), ints, strings, btyes and more. These were done as users reported reproducible performance problems." 1. typo "btyes" 2. It is also the first PyPy release to take advantage of "type freezing", a specialization for objects with type stable attributes (described in https://dl.acm.org/doi/abs/10.1145/3368826.3377907) which greatly improves efficiency of numerically oriented code using a lot of small Python objects containing numbers. With PyPy 7.3.4, a pure Python implementation of the NBody problem (bench_purepy_Particle.py, see https://github.com/paugier/nbabel#table-of-codes) is basically as fast as our implementations in C++ and Fortran. Pierre ----- Mail original ----- > De: "Matti Picus" > ?: "pypy-dev" > Envoy?: Vendredi 19 Mars 2021 15:31:33 > Objet: [pypy-dev] PyPy 7.3.4rc1 tarballs are up > The first release candidate for PyPy7.3.4 is up at > https://downloads.python.org/pypy/. People using cibuildwheel or github > actions to test PyPy should be able to use this in their test runs. > > > The sha256 checksums can be found in the PR to the pypy.org web site at > > > https://github.com/pypy/pypy.org/pull/13/files#diff-622ff636d3511de2e408d8b4134b4b825743385cb6c8d9bbadf6dc5b0c887f0eR491 > > > or via the preview at > > > https://deploy-preview-13--keen-mestorf-442210.netlify.app/download_advanced.html#checksums > > > The draft release note is up at > > > https://doc.pypy.org/en/latest/release-v7.3.4.html > > > This is an incrementally better release: there are no ground-breaking > changes. However there are a number of JIT preformance enhancements, and > overall better compatibility with CPython. > > > Please try it out and report issues on IRC or on the issue tracker. > > > Matti > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From lql9705 at gmail.com Mon Mar 29 20:35:54 2021 From: lql9705 at gmail.com (Kevin Lee) Date: Tue, 30 Mar 2021 08:35:54 +0800 Subject: [pypy-dev] A little problem in unicodeobject.py Message-ID: Dear developers of PyPy: I found that there might be a little bug in `unicodeobject.py`. Our project relies on `sys.setdefautencoding('utf-8')` to handle string encoding things(we use Python2.7), which is not a good practice though, when we upgrade pypy to 7.1.0 and any version newer than it, our project stops working with tons of UnicodeEncodeError exceptions. There is already an issue 3329 described this problem, I left my comment below it which provided our solution to this problem. But I am not sure our solution is the correct way, and I tried to make a merge request but it then said that I don't have write permission, so I decided to write an email. I'm looking forward to hearing from you guys soon. Best regards, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: