From matti.picus at gmail.com Tue Oct 1 13:01:46 2019 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 1 Oct 2019 20:01:46 +0300 Subject: [pypy-dev] Migration from Bitbucket to ??? In-Reply-To: <427264973.24684183.1569871529602.JavaMail.zimbra@univ-grenoble-alpes.fr> References: <427264973.24684183.1569871529602.JavaMail.zimbra@univ-grenoble-alpes.fr> Message-ID: On 30/9/19 10:25 pm, PIERRE AUGIER wrote: > Hi, > > I posted this message in https://bitbucket.org/pypy/pypy/issues/3082 but apparently, it's better to post it in this list, so here it is: > > As explained here https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket, Mercurial features and repositories will be officially removed from Bitbucket and its API on June 1, 2020. > > I guess you thought about what to do for PyPy repository and PyPy development. > > I am also a Mercurial user and I am facing the same problem for some repositories. Thus, I am very interested to know what you plan to do. > > In particular, I am looking for a good forge for the Transonic project (https://transonic.readthedocs.io), which by the way is compatible with PyPy3.6 (currently only the nightly builds) and could help a wider adoption of PyPy3 for projects using the Python scientific stack (just by accelerating the numerical kernels using Numpy and let PyPy accelerate pure Python code). > > I follow the Heptapod project https://heptapod.net/ (a friendly fork of GitLab to bring very good Mercurial support). It seems to me that it starts to be nearly production ready and that it would nicely fit your workflow with Mercurial. > > Of course, it would be very convenient if a common instance (free to use for simple users until a certain point) could be setup (something somehow similar to Github, but with Mercurial support). Even for PyPy, it would be good in terms of visibility. > > I also heard that your repository would have to be fixed to be able to use Heptapod. > > What is the current status of these issues for PyPy? Have you already decided what to do? > > Best regards, > Pierre > > -- > Pierre Augier - CR CNRS http://www.legi.grenoble-inp.fr > LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels > BP53, 38041 Grenoble Cedex, France tel:+33.4.56.52.86.16 Hi Pierre. We have not formulated an official PyPy strategy. We do have some general guidelines and thoughts that we put up here https://etherpad.net/p/Alternatives_to_hosting_PyPy_on_BitBucket, but no conclusions yet. Most of us are hopeful heptopod will provide a viable solution. They seem to have fixed the blockers so our repo can be used as-is, I would like to see some kind of mutual support group, are there other projects that would very much prefer to stay on mercurial? Matti From tkadm30 at yandex.com Tue Oct 1 18:57:57 2019 From: tkadm30 at yandex.com (Jack Bortone) Date: Tue, 1 Oct 2019 18:57:57 -0400 Subject: [pypy-dev] Migration from Bitbucket to ??? In-Reply-To: <427264973.24684183.1569871529602.JavaMail.zimbra@univ-grenoble-alpes.fr> References: <427264973.24684183.1569871529602.JavaMail.zimbra@univ-grenoble-alpes.fr> Message-ID: On 2019-09-30 3:25 p.m., PIERRE AUGIER wrote: > Hi, I posted this message in > https://bitbucket.org/pypy/pypy/issues/3082 but apparently, it's > better to post it in this list, so here it is: As explained here > https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket, > Mercurial features and repositories will be officially removed from > Bitbucket and its API on June 1, 2020. I guess you thought about what > to do for PyPy repository and PyPy development. I am also a Mercurial > user and I am facing the same problem for some repositories. Thus, I > am very interested to know what you plan to do. In particular, I am > looking for a good forge for the Transonic project > (https://transonic.readthedocs.io), which by the way is compatible > with PyPy3.6 (currently only the nightly builds) and could help a > wider adoption of PyPy3 for projects using the Python scientific stack > (just by accelerating the numerical kernels using Numpy and let PyPy > accelerate pure Python code). I follow the Heptapod project > https://heptapod.net/ (a friendly fork of GitLab to bring very good > Mercurial support). It seems to me that it starts to be nearly > production ready and that it would nicely fit your workflow with > Mercurial. Of course, it would be very convenient if a common instance > (free to use for simple users until a certain point) could be setup > (something somehow similar to Github, but with Mercurial support). > Even for PyPy, it would be good in terms of visibility. I also heard > that your repository would have to be fixed to be able to use > Heptapod. What is the current status of these issues for PyPy? Have > you already decided what to do? Best regards, Pierre -- Pierre Augier > - CR CNRS http://www.legi.grenoble-inp.fr LEGI (UMR 5519) Laboratoire > des Ecoulements Geophysiques et Industriels BP53, 38041 Grenoble > Cedex, France tel:+33.4.56.52.86.16 Good evening Pierre, As far we're concerned and currently having all our open projects still hosted on Bitbucket.org we decided to deploy our own mercurial based repository server with uWSGI and nginx. We also think Mercurial is a great SCM which simplifies a lot the pain of having to use git for every day SCM management. In our opinion, it is a unfortunate decision that Bitbucket has decided to migrate from Mercurial to git since it is a very stable and user friendly SCM so we are going to close our account eventually. Best regards, Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Oct 2 08:23:18 2019 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 2 Oct 2019 15:23:18 +0300 Subject: [pypy-dev] P:yPy 7.2 release candidates uploaded, please try them out Message-ID: I uploaded 7.2.0rc tarballs to https://bitbucket.org/pypy/pypy/downloads/ Here are the sha256 hashes: 00b0042d904f1fa1aa5018f5411b8f1715bb7a72ecb5a8193ca8d45dc5754dd6 pypy2.7-v7.2.0rc0-aarch64.tar.bz2 3ab83172ebce4bba3e45c2f583d9419cdb7224cec6eccbb77a49d9377f38cf53 pypy2.7-v7.2.0rc0-linux32.tar.bz2 4e19d649c7442fafe7e342269c3b6da63ebabbb5ee7f5bb0c1a90e351894dff3 pypy2.7-v7.2.0rc0-linux64.tar.bz2 13cbbb8dd7a4cd148b074920b0a69e36828cb14d5a0bf01c0708ec7b3368d5f7 pypy2.7-v7.2.0rc0-osx64.tar.bz2 de8cdce8a35ae9cb48750045e864775752bea30c28096f7209819006d54814f0 pypy2.7-v7.2.0rc0-s390x.tar.bz2 616d66ee4f1020289c57bdc753987d89c0814529196c86f9aac5a51fea05d3f8 pypy2.7-v7.2.0rc0-src.tar.bz2 734a6f4ea1921f533f6ddfb0d07567a9c53389f99f633a965eb4b3ed67ee50ad pypy2.7-v7.2.0rc0-src.zip 9ce3049d40ed136d63ebe5e051899f897aee01dad5a1ffe9bd676fb4c3c6f7ce pypy2.7-v7.2.0rc0-win32.zip 76e16ff998bb1e624942546d5657854eed032fbc27a23d5a7d4eafeada13b059 pypy3.6-v7.2.0rc1-aarch64.tar.bz2 c0505ca984f39cc276584e836e073e7521c4c601890a0634efefff96cbb10bc6 pypy3.6-v7.2.0rc1-linux32.tar.bz2 6046ca5b4adfe7d82cd08128d09bc8cfc808e6da946f47eadf8a88014bed9a34 pypy3.6-v7.2.0rc1-linux64.tar.bz2 b5f28fbb14a3774d7367b5af4f2cf971261c080320f6d6bb91c129ae8e3ccf10 pypy3.6-v7.2.0rc1-osx64.tar.bz2 ecc6bf68b1e2a1d37bdc2534b561bdfcb2a091a329b00f50417aca6456e54847 pypy3.6-v7.2.0rc1-s390x.tar.bz2 810ab3bc7252742c2d18cd07c3ea78f7afedd1693c2195b2f396986809a0c6a6 pypy3.6-v7.2.0rc1-src.tar.bz2 dfb502138069ef2c3865981cc12414052e79b3329791fae8130af232a4f55f42 pypy3.6-v7.2.0rc1-src.zip b08cde67843657497e39c7f6c60774677f43b736e91f79d8707c75652c928c2f pypy3.6-v7.2.0rc1-win32.zip The python3 ones are rc1 since I did not want to move a tag, and I improperly tagged rc0. Please try them out. I already know that we will need to add 38ede7e5cb5a for very large JSON maps, let me know if there are more. Matti From horvath.kristof.attila at gmail.com Wed Oct 9 09:20:50 2019 From: horvath.kristof.attila at gmail.com (=?UTF-8?B?S3Jpc3TDs2YgSG9ydsOhdGg=?=) Date: Wed, 9 Oct 2019 15:20:50 +0200 Subject: [pypy-dev] Windows 10 error Message-ID: Hello, I downloaded the Windows binary, but pypy3.exe fails on startup with error code 0xc000007b (I tried version 3.6 and 3.5, too). I use Windows 10 x64, can it cause the problem? I read in the building instructions that only a 32-bit version available on Windows at this time, but it should be work on a 64-bit system, right? I couldn't find any useful information about the problem on the internet, have you got any idea what's wrong? Thank you for your help. Regards, Krist?f Horv?th -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Wed Oct 9 10:28:47 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 9 Oct 2019 16:28:47 +0200 Subject: [pypy-dev] Windows 10 error In-Reply-To: References: Message-ID: Hi, On Wed, 9 Oct 2019 at 16:16, Krist?f Horv?th wrote: > I downloaded the Windows binary, but pypy3.exe fails on startup with error code 0xc000007b (I tried version 3.6 and 3.5, too). I use Windows 10 x64, can it cause the problem? I read in the building instructions that only a 32-bit version available on Windows at this time, but it should be work on a 64-bit system, right? Yes. Assuming (randomly) that this is because of a missing dependencies, can you try: 1. Install http://www.microsoft.com/en-us/download/details.aspx?id=5582 2. If it doesn't help, check that you have the following files, e.g. in C:\Windows\System32: ADVAPI32.dll WS2_32.dll VCRUNTIME140.dll Armin From armin.rigo at gmail.com Wed Oct 9 10:35:10 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 9 Oct 2019 16:35:10 +0200 Subject: [pypy-dev] Windows 10 error In-Reply-To: References: Message-ID: Hi again, On Wed, 9 Oct 2019 at 16:28, Armin Rigo wrote: > 1. Install http://www.microsoft.com/en-us/download/details.aspx?id=5582 Oops, this link is now broken. Thanks MS. Here is another page that mentions the right VCRuntime140.dll in the question title: https://answers.microsoft.com/en-us/windows/forum/windows_10-update/vcruntime140dll/b1a15b0e-e389-41a3-bc49-f4fc85bac575 https://www.microsoft.com/en-us/download/details.aspx?id=52685 You need the 32-bit version. Armin From armin.rigo at gmail.com Wed Oct 9 11:54:48 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 9 Oct 2019 17:54:48 +0200 Subject: [pypy-dev] Windows 10 error In-Reply-To: References: Message-ID: Hi, On Wed, 9 Oct 2019 at 16:45, Krist?f Horv?th wrote: > Thank you for your reply. The listed dll-s already were parts of my system, but installing Visual C++ Redistributable solved the problem, so it works now. Perhaps, you could write a note about the dependencies onto your installing page. Thanks for the confirmation! I've updated the page http://pypy.org/download.html. A bient?t, Armin From matti.picus at gmail.com Fri Oct 11 01:30:12 2019 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 11 Oct 2019 08:30:12 +0300 Subject: [pypy-dev] P:yPy 7.2 release candidates uploaded, please try them out Message-ID: <7ae184e4-2c4c-4d49-2dd4-aa43528cc9fb@gmail.com> I uploaded 7.2.0rc2 tarballs to https://bitbucket.org/pypy/pypy/downloads/ Here are the sha256 hashes: 738df122fd3f202397f6efea641e952214197541f2a0e79112767cd3cfb6eb16 pypy2.7-v7.2.0rc2-aarch64.tar.bz2 53685a718ffb651a82a3fa6de89c8c4229a4a38b358a26100af688aa2eb3e32b pypy2.7-v7.2.0rc2-linux32.tar.bz2 bf8c9a0a0790f77050049c05ac2e66fe09b03f714a613b628bfee91db7e13fb7 pypy2.7-v7.2.0rc2-linux64.tar.bz2 6a9018f38046a51ebe466a5ecde57480b657bf55757669bd3ccb9dabc97652e2 pypy2.7-v7.2.0rc2-osx64.tar.bz2 448ffe6c14df01a0c0a98571731438c78e6165a3959d711e7e5233ba9a44ef81 pypy2.7-v7.2.0rc2-s390x.tar.bz2 4691c4d6566fdcc68ccbb77c1456f004b8749adf397c167472d8b12337291d37 pypy2.7-v7.2.0rc2-src.tar.bz2 2064b6ebd83b809a2a14c34421b63fe6c2a91572502a22e3db06a6dd40bb73c9 pypy2.7-v7.2.0rc2-src.zip 551e8d8cecf78b3892bf0c93fcb24835b9dfb9c0a6a15c37e65f7e775fab7d1b pypy2.7-v7.2.0rc2-win32.zip 5b37cfd78291b336923e87e369c5629390e2143a74f5dee10463ddeac2f35cc8 pypy3.6-v7.2.0rc2-aarch64.tar.bz2 cf0d3421ba0d08f472358d7601fdafce0239b46666fe829b28c77a178c01a719 pypy3.6-v7.2.0rc2-linux32.tar.bz2 96d7274e40cd1e1627b3c29dfda80cb1a1fe22ec3bbc45813ae42b60903b853e pypy3.6-v7.2.0rc2-linux64.tar.bz2 3c2339fc29a6aeb691c52865f44880b30462abd376a4cdf5bbda169594a3bc0d pypy3.6-v7.2.0rc2-osx64.tar.bz2 c4c04c49e707e14c9dbf26847fe2b6ac17271211582ca4f9c8f4785bde2aee51 pypy3.6-v7.2.0rc2-s390x.tar.bz2 a663b6c402b07d3f1598af1836f0573861cba0ce37590b7e3035e5ef8f2186ea pypy3.6-v7.2.0rc2-src.tar.bz2 6be68e337a71152d654da840ca21d0d05a12d6b7727303e6c36b69e93e2bcc76 pypy3.6-v7.2.0rc2-src.zip d1d043d83bfbcb68b94bccbd4eaced8edb285d7f4f52323af481f0f2007e1cdb pypy3.6-v7.2.0rc2-win32.zip If there are no problems I will repackage this commit as the release in a few days Matti From kolecdav at fel.cvut.cz Wed Oct 16 09:34:33 2019 From: kolecdav at fel.cvut.cz (=?utf-8?Q?David_Kole=C4=8Dk=C3=A1=C5=99?=) Date: Wed, 16 Oct 2019 15:34:33 +0200 Subject: [pypy-dev] contributing to PyPy Message-ID: <46tYR33STRzpCY4@mail.python.org> Hello, I would like to ask You, if there is some chance to help with the PyPy project. I am studying MS Computer Sciences at CTU FEE, Prague. Thank You for answer David Koleckar -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Wed Oct 16 17:01:43 2019 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 16 Oct 2019 14:01:43 -0700 (PDT) Subject: [pypy-dev] contributing to PyPy In-Reply-To: <46tYR33STRzpCY4@mail.python.org> References: <46tYR33STRzpCY4@mail.python.org> Message-ID: David, > I would like to ask You, if there is some chance to help with the PyPy > project. I am studying MS Computer Sciences at CTU FEE, Prague. there is this: https://doc.pypy.org/en/latest/contributing.html and in particular the "Your first contribution" section. Myself, I have only worked on one module, cppyy, and currently support on the PyPy side is behind CPython, which I hope to fix in the near future. Description/documentation of cppyy is here: https://cppyy.readthedocs.io/en/latest/ If you have expertise and interest in that direction, LMK. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From mgorny at gentoo.org Sat Oct 19 16:55:36 2019 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Sat, 19 Oct 2019 22:55:36 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? Message-ID: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> Hello, I've noticed that the compiled module suffix keeps changing between PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72- *.so (also .pyc). However, this is a bit surprising to me given that for PyPy2 it's still at .pypy-41.so. Is the bytecode generated by successive PyPy3 releases really incompatible between them? Or are the suffix changes only incidental? They cause quite some trouble for us, since they make it necessary to recompile installed modules on Gentoo, and PyPy's overzealous compiling causes access violations for our users. TIA for any help. -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Sun Oct 20 00:32:05 2019 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 20 Oct 2019 07:32:05 +0300 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> Message-ID: On 19/10/19 11:55 pm, Micha? G?rny wrote: > Hello, > > I've noticed that the compiled module suffix keeps changing between > PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72- > *.so (also .pyc). However, this is a bit surprising to me given that > for PyPy2 it's still at .pypy-41.so. > > Is the bytecode generated by successive PyPy3 releases really > incompatible between them? Or are the suffix changes only incidental? > They cause quite some trouble for us, since they make it necessary to > recompile installed modules on Gentoo, and PyPy's overzealous compiling > causes access violations for our users. > > TIA for any help. For c-extensions the major-minor version becomes the PEP 425 ABI tag https://www.python.org/dev/peps/pep-0425/#abi-tag. My operating assumption is that any change in the ABI requires a change in that field. Our support for the massive CPython C-API changes between versions, which changes the ABI. In particular, between 7.1 and 7.2 the PyDateTime_CAPI structure size changed, and we changed the order of fields in Py_Buffer, as well as adding many missing functions and macros. The complete log is here http://doc.pypy.org/en/latest/release-v7.2.0.html#c-api-cpyext-and-c-extensions for both versions and here http://doc.pypy.org/en/latest/release-v7.2.0.html#python-3-6-c-api for the pypy3.6 specific ones. The fact that the PyPy2 ABI tag did **not** change is most likely a bug in the release process. I think there will be cases where pypy2-v7.1.1 and pypy2-v7.2.0 c-extension modules will be slightly incompatible with eachother, although a cursory test of mixing NumPy across versions (building on 7.2, copying into 7.1) seems to pass "np.test()". Perhaps a better test would be to mix the pygolang c-extension modules, which seem to stress-test the C-API more extensively given the number of issues it exposes. The pyc files should always be rebuilt in each python environment, so I am not sure what problems could be caused by bumping the ABI tag. Does Gentoo somehow mix the byte-compiled pyc files across versions? I am not sure what you mean by "compiling causes access violations for our users", could you point to a discussion of the problem? Matti From mgorny at gentoo.org Sun Oct 20 03:01:54 2019 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Sun, 20 Oct 2019 09:01:54 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> Message-ID: <810d1e0f83c301b9a1a7ea3eba18e0d1cac254b7.camel@gentoo.org> On Sun, 2019-10-20 at 07:32 +0300, Matti Picus wrote: > On 19/10/19 11:55 pm, Micha? G?rny wrote: > > Hello, > > > > I've noticed that the compiled module suffix keeps changing between > > PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72- > > *.so (also .pyc). However, this is a bit surprising to me given that > > for PyPy2 it's still at .pypy-41.so. > > > > Is the bytecode generated by successive PyPy3 releases really > > incompatible between them? Or are the suffix changes only incidental? > > They cause quite some trouble for us, since they make it necessary to > > recompile installed modules on Gentoo, and PyPy's overzealous compiling > > causes access violations for our users. > > > > TIA for any help. > For c-extensions the major-minor version becomes the PEP 425 ABI tag > https://www.python.org/dev/peps/pep-0425/#abi-tag. My operating > assumption is that any change in the ABI requires a change in that > field. Our support for the massive CPython C-API changes between > versions, which changes the ABI. In particular, between 7.1 and 7.2 the > PyDateTime_CAPI structure size changed, and we changed the order of > fields in Py_Buffer, as well as adding many missing functions and > macros. The complete log is here > http://doc.pypy.org/en/latest/release-v7.2.0.html#c-api-cpyext-and-c-extensions > for both versions and here > http://doc.pypy.org/en/latest/release-v7.2.0.html#python-3-6-c-api for > the pypy3.6 specific ones. > > > The fact that the PyPy2 ABI tag did **not** change is most likely a bug > in the release process. I think there will be cases where pypy2-v7.1.1 > and pypy2-v7.2.0 c-extension modules will be slightly incompatible with > eachother, although a cursory test of mixing NumPy across versions > (building on 7.2, copying into 7.1) seems to pass "np.test()". Perhaps a > better test would be to mix the pygolang c-extension modules, which seem > to stress-test the C-API more extensively given the number of issues it > exposes. Thanks for the explanation. This answers my question. Do you need me to file a bug for the ABI tag not changing in PyPy2, or will you take it from here? > The pyc files should always be rebuilt in each python environment, so I > am not sure what problems could be caused by bumping the ABI tag. Does > Gentoo somehow mix the byte-compiled pyc files across versions? > > > I am not sure what you mean by "compiling causes access violations for > our users", could you point to a discussion of the problem? > It's related to circular dependencies between packages. For example, when rebuild setuptools for the new version, it tries to load its plugins and byte-compile them as well. Since they don't belong to the setuptools package, our PM catches that as illegal access. It's a generic problem with Python, not something you need to worry about. I've just been pointed out that we already have a hack for it, I just need to add PyPy3 to it. -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Sun Oct 20 03:46:57 2019 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 20 Oct 2019 10:46:57 +0300 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: <810d1e0f83c301b9a1a7ea3eba18e0d1cac254b7.camel@gentoo.org> References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <810d1e0f83c301b9a1a7ea3eba18e0d1cac254b7.camel@gentoo.org> Message-ID: <3a47c5ac-1fe1-315b-1657-4842cd595b36@gmail.com> On 20/10/19 10:01 am, Micha? G?rny wrote: > Thanks for the explanation. This answers my question. Do you need me > to file a bug for the ABI tag not changing in PyPy2, or will you take it > from here? > I would like to confirm that in fact there is an issue: that the c-extension shared objects are incompatible. I am not completely convinced this is the case, at least my experimentation with NumPy proved it indeed is *not* the case for PyPy2. I am open to hearing opinions from others. Is there a concensus around whether we do need to change the ABI designation? I think this would also require recompilation of any CFFI shared objects on PyPy2. Matti From armin.rigo at gmail.com Sun Oct 20 16:21:39 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Sun, 20 Oct 2019 22:21:39 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: <3a47c5ac-1fe1-315b-1657-4842cd595b36@gmail.com> References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <810d1e0f83c301b9a1a7ea3eba18e0d1cac254b7.camel@gentoo.org> <3a47c5ac-1fe1-315b-1657-4842cd595b36@gmail.com> Message-ID: Hi Matti, On Sun, 20 Oct 2019 at 09:47, Matti Picus wrote: > I would like to confirm that in fact there is an issue: that the > c-extension shared objects are incompatible. I am not completely > convinced this is the case, at least my experimentation with NumPy > proved it indeed is *not* the case for PyPy2. I am open to hearing > opinions from others. Is there a concensus around whether we do need to > change the ABI designation? I think this would also require > recompilation of any CFFI shared objects on PyPy2. In PyPy2 there are two different numbers: the version in the ".pypy-XY.so" extension, and the internal version in the ".pyc" files. In PyPy3 the ".pyc" files have grown to ".pypy-XY.pyc". (This is confusing because if you translate PyPy3.6 and the in-progress PyPy3.7 then they'll try to use the same ".pypy-XY.pyc" extension, even though the internal bytecode version in that file is different.) If we want a single number that changes mostly every release, then we're doing the right thing. If instead we prefer to keep several more precise numbers, we should use different numbers for (a) the C extensions; (b) the .pyc files; and even (c) the cffi modules. As far as I understand, the problem with doing that is that people used to (and code used on) CPython are not really ready to handle this situation. As for the precise question you're asking, "do we need to change the ABI designation in PyPy2", the answer is yes, imho: we should change it as soon as we break the ABI, even if only in a corner case that doesn't concern most C extensions... A bient?t, Armin. From matti.picus at gmail.com Tue Oct 22 09:34:36 2019 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 22 Oct 2019 16:34:36 +0300 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <810d1e0f83c301b9a1a7ea3eba18e0d1cac254b7.camel@gentoo.org> <3a47c5ac-1fe1-315b-1657-4842cd595b36@gmail.com> Message-ID: On 20/10/19 11:21 pm, Armin Rigo wrote: > Hi Matti, > if you translate PyPy3.6 and the in-progress PyPy3.7 > then they'll try to use the same ".pypy-XY.pyc" extension, even though > the internal bytecode version in that file is different.) I see CPython uses the python {major}{minor} version: "example.cpython-36.pyc". We should probably change our convention to do the same. Any idea where that happens? > As for the precise question you're asking, "do we need to change the > ABI designation in PyPy2", the answer is yes, imho: we should change > it as soon as we break the ABI, even if only in a corner case that > doesn't concern most C extensions... > > > A bient?t, > > Armin. The code in question is in pypy/module/imp/importing.py, and has a comment # this used to change for every minor version, but no longer does: there # is little point any more, as the so's tend to be cross-version- # compatible, more so than between various versions of CPython. Be # careful if we need to update it again: it is now used for both cpyext # and cffi so's.? If we do have to update it, we'd likely need a way to # split the two usages again. #DEFAULT_SOABI = 'pypy-%d%d' % PYPY_VERSION[:2] DEFAULT_SOABI = 'pypy-41' So do we update it across the board for each change in the cpyext ABI? Matti From armin.rigo at gmail.com Wed Oct 23 10:32:02 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 23 Oct 2019 16:32:02 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <810d1e0f83c301b9a1a7ea3eba18e0d1cac254b7.camel@gentoo.org> <3a47c5ac-1fe1-315b-1657-4842cd595b36@gmail.com> Message-ID: Hi Matti, On Tue, 22 Oct 2019 at 15:34, Matti Picus wrote: > #DEFAULT_SOABI = 'pypy-%d%d' % PYPY_VERSION[:2] > DEFAULT_SOABI = 'pypy-41' > > So do we update it across the board for each change in the cpyext ABI? No, my point was that if we want to do that we should first split the usages, and only update the version used for C extension modules. In other words we should not update it for cffi modules (which is unlikely to ever change, and I can check but I think the same .so works for pypy2 and pypy3, so maybe a version number is not needed at all); and also not for .pyc files (which should just be "pypy-36" for pypy3 implementing python 3.6, and if at some point we really want to add a new bytecode, then well, we'll think again, I suppose). A bient?t, Armin. From armin.rigo at gmail.com Wed Oct 23 10:46:52 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Wed, 23 Oct 2019 16:46:52 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <810d1e0f83c301b9a1a7ea3eba18e0d1cac254b7.camel@gentoo.org> <3a47c5ac-1fe1-315b-1657-4842cd595b36@gmail.com> Message-ID: Hi again, On Wed, 23 Oct 2019 at 16:32, Armin Rigo wrote: > (...) In > other words we should not update it for cffi modules (which is > unlikely to ever change, and I can check but I think the same .so > works for pypy2 and pypy3, so maybe a version number is not needed at > all); Yes, I think that's the case. The .so for cffi should be almost entirely the same on pypy2 and pypy3, with one minor difference that turns out not to matter. (The module exports a function _cffi_pypyinit__foo() that is declared to return void on pypy2, but "PyObject *" on pypy3---where it returns NULL and the actual return value is never checked. We do it that way because we're reusing the convenient macro PyMODINIT_FUNC from Python.h.) A bient?t, Armin. From matti.picus at gmail.com Fri Oct 25 03:23:09 2019 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 25 Oct 2019 10:23:09 +0300 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> Message-ID: On 19/10/19 11:55 pm, Micha? G?rny wrote: > Hello, > > I've noticed that the compiled module suffix keeps changing between > PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72- > *.so (also .pyc). However, this is a bit surprising to me given that > for PyPy2 it's still at .pypy-41.so. > > Is the bytecode generated by successive PyPy3 releases really > incompatible between them? Or are the suffix changes only incidental? > They cause quite some trouble for us, since they make it necessary to > recompile installed modules on Gentoo, and PyPy's overzealous compiling > causes access violations for our users. > > TIA for any help. > I committed changes that: - on py3.6 (for python 3.6, 3.7 and up) change the *.pyc file name to follow the cpython spec: filename.pypy-36.pyc - on default (for python2) change the DEFAULT_SOABI to track the pypy_version; so's will now be named .pypy-72.so (on py3.6 they are named filename.pypy3-72-x86_64-linux-gnu.so so they will not clash) For CFFI, we discussed a pypy-specific "stable-api" extension to mirror the cpython3 "abi3" tag, the idea still needs to be fleshed out and implemented. The reasoning behind the changes to the pyc filename and so filenames is explained earlier in this mail thread, I will not repeat them here but if I was mistaken please help me get it right. The changes, if not reverted, will be part of the next release cycle. Matti From mgorny at gentoo.org Fri Oct 25 03:37:00 2019 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Fri, 25 Oct 2019 09:37:00 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> Message-ID: On Fri, 2019-10-25 at 10:23 +0300, Matti Picus wrote: > On 19/10/19 11:55 pm, Micha? G?rny wrote: > > Hello, > > > > I've noticed that the compiled module suffix keeps changing between > > PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72- > > *.so (also .pyc). However, this is a bit surprising to me given that > > for PyPy2 it's still at .pypy-41.so. > > > > Is the bytecode generated by successive PyPy3 releases really > > incompatible between them? Or are the suffix changes only incidental? > > They cause quite some trouble for us, since they make it necessary to > > recompile installed modules on Gentoo, and PyPy's overzealous compiling > > causes access violations for our users. > > > > TIA for any help. > > > I committed changes that: > > - on py3.6 (for python 3.6, 3.7 and up) change the *.pyc file name to > follow the cpython spec: filename.pypy-36.pyc > > - on default (for python2) change the DEFAULT_SOABI to track the > pypy_version; so's will now be named .pypy-72.so (on py3.6 they are > named filename.pypy3-72-x86_64-linux-gnu.so so they will not clash) Shouldn't this include the 3.6/3.7 version as well? > For CFFI, we discussed a pypy-specific "stable-api" extension to mirror > the cpython3 "abi3" tag, the idea still needs to be fleshed out and > implemented. > > The reasoning behind the changes to the pyc filename and so filenames is > explained earlier in this mail thread, I will not repeat them here but > if I was mistaken please help me get it right. > > The changes, if not reverted, will be part of the next release cycle. > Thanks! -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Fri Oct 25 03:49:19 2019 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 25 Oct 2019 10:49:19 +0300 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> Message-ID: <15f44e6e-ecc5-b840-1d9c-cfeaecd76319@gmail.com> On 25/10/19 10:37 am, Micha? G?rny wrote: >> - on default (for python2) change the DEFAULT_SOABI to track the >> pypy_version; so's will now be named .pypy-72.so (on py3.6 they are >> named filename.pypy3-72-x86_64-linux-gnu.so so they will not clash) > Shouldn't this include the 3.6/3.7 version as well? > Yes, actually, thanks. They should be named {python tag}-{abi tag}-{platform tag}, so pypy36-pp72-x86_64-linux-gnu.so. I will make sure the python3.6/python3.7 so names do not overlap before we release a python3.7 alpha. Matti From matti.picus at gmail.com Fri Oct 25 04:21:00 2019 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 25 Oct 2019 11:21:00 +0300 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: <15f44e6e-ecc5-b840-1d9c-cfeaecd76319@gmail.com> References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <15f44e6e-ecc5-b840-1d9c-cfeaecd76319@gmail.com> Message-ID: <3836f1ed-e906-9f4c-3b55-2e8c1ee67736@gmail.com> On 25/10/19 10:49 am, Matti Picus wrote: > > On 25/10/19 10:37 am, Micha? G?rny wrote: >>> - on default (for python2) change the DEFAULT_SOABI to track the >>> pypy_version; so's will now be named .pypy-72.so (on py3.6 they are >>> named filename.pypy3-72-x86_64-linux-gnu.so so they will not clash) >> Shouldn't this include the 3.6/3.7 version as well? >> > > Yes, actually, thanks. They should be named {python tag}-{abi > tag}-{platform tag}, so pypy36-pp72-x86_64-linux-gnu.so. I will make > sure the python3.6/python3.7 so names do not overlap before we release > a python3.7 alpha. > > Matti > Needs more thought. The changes in the C-API are reflected in the platform tag: 71 is incompatible (perhaps only slightly) with 72. What breaking changes are there from the perspective of a C-API module between python 3.6 to 3.7, 3.8, 3.9? Matti From armin.rigo at gmail.com Fri Oct 25 08:09:14 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Fri, 25 Oct 2019 14:09:14 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: <3836f1ed-e906-9f4c-3b55-2e8c1ee67736@gmail.com> References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <15f44e6e-ecc5-b840-1d9c-cfeaecd76319@gmail.com> <3836f1ed-e906-9f4c-3b55-2e8c1ee67736@gmail.com> Message-ID: Hi Matti, On Fri, 25 Oct 2019 at 10:21, Matti Picus wrote: > > > On 25/10/19 10:49 am, Matti Picus wrote: > > Yes, actually, thanks. They should be named {python tag}-{abi > > tag}-{platform tag}, so pypy36-pp72-x86_64-linux-gnu.so. I will make > > sure the python3.6/python3.7 so names do not overlap before we release > > a python3.7 alpha. > > Needs more thought. The changes in the C-API are reflected in the > platform tag: 71 is incompatible (perhaps only slightly) with 72. What > breaking changes are there from the perspective of a C-API module > between python 3.6 to 3.7, 3.8, 3.9? The C module itself may contain "#if PY_VERSION_HEX >= 0x03070000" or similar, in order to compile some feature (or work around some issue) that is only available on CPython 3.{N} but no 3.{N-1}. So I think it's a good idea to include both the CPython and the PyPy version in the name. A bient?t, Armin. From matti.picus at gmail.com Sun Oct 27 05:32:53 2019 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 27 Oct 2019 11:32:53 +0200 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <15f44e6e-ecc5-b840-1d9c-cfeaecd76319@gmail.com> <3836f1ed-e906-9f4c-3b55-2e8c1ee67736@gmail.com> Message-ID: <7f42b7d1-0bef-7fa8-476d-eca867a3d921@gmail.com> On 25/10/19 3:09 pm, Armin Rigo wrote: > Hi Matti, > > On Fri, 25 Oct 2019 at 10:21, Matti Picus wrote: >> >> On 25/10/19 10:49 am, Matti Picus wrote: >>> Yes, actually, thanks. They should be named {python tag}-{abi >>> tag}-{platform tag}, so pypy36-pp72-x86_64-linux-gnu.so. I will make >>> sure the python3.6/python3.7 so names do not overlap before we release >>> a python3.7 alpha. >> Needs more thought. The changes in the C-API are reflected in the >> platform tag: 71 is incompatible (perhaps only slightly) with 72. What >> breaking changes are there from the perspective of a C-API module >> between python 3.6 to 3.7, 3.8, 3.9? > The C module itself may contain "#if PY_VERSION_HEX >= 0x03070000" or > similar, in order to compile some feature (or work around some issue) > that is only available on CPython 3.{N} but no 3.{N-1}. So I think > it's a good idea to include both the CPython and the PyPy version in > the name. > > > A bient?t, > > Armin. Would this be considered a major API breaking change or only a revision change? Would we need to change to pypy 8.0 (i.e. pypy36-pp80-x86_64-linux-gnu.so), or can we stay with pypy 7.3 (i.e., pypy36-pp73-x86_64-linux-gnu.so)? In any case, wheels made for pypy before this change would not be compatible with ones after it. Matti From armin.rigo at gmail.com Tue Oct 29 04:33:41 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Tue, 29 Oct 2019 09:33:41 +0100 Subject: [pypy-dev] PyPy3: is bytecode really incompatible between releases? In-Reply-To: <7f42b7d1-0bef-7fa8-476d-eca867a3d921@gmail.com> References: <7e66fac8c52f6121fe4bf8aa6bcbc0973ec735d9.camel@gentoo.org> <15f44e6e-ecc5-b840-1d9c-cfeaecd76319@gmail.com> <3836f1ed-e906-9f4c-3b55-2e8c1ee67736@gmail.com> <7f42b7d1-0bef-7fa8-476d-eca867a3d921@gmail.com> Message-ID: Hi Matti, On Sun, 27 Oct 2019 at 10:33, Matti Picus wrote: > Would this be considered a major API breaking change or only a revision > change? Would we need to change to pypy 8.0 (i.e. > pypy36-pp80-x86_64-linux-gnu.so), or can we stay with pypy 7.3 (i.e., > pypy36-pp73-x86_64-linux-gnu.so)? In any case, wheels made for pypy > before this change would not be compatible with ones after it. I like to think that major versions of PyPy should also indicate that we did some major work in other areas, like the JIT compiler, the json decoder, etc. etc. The question of whether the next version should be called "7.3" or "8.0" should weight on that IMHO. It should not depend *only* on whether we broke the API inside cpyext. That means cpyext needs to have its own way to tell that the API broke; for example, it could use file names "pypy36-pp#-x86_64-linux-gnu.so" with the "#" being the API version number. Something like that. Maybe just an increasing number starting at 42 (as the number following the "pypy41" we use so far; unrelated to the meaning of life!) A bient?t, Armin.