From gyromagnetic at gmail.com Tue Sep 1 17:58:13 2020 From: gyromagnetic at gmail.com (gyro funch) Date: Tue, 1 Sep 2020 15:58:13 -0600 Subject: [pypy-dev] help building pypy Message-ID: Hi, I am interested in building pypy 3.6 on Ubuntu 20.04 LTS. I downloaded the prebuilt binaries for pypy 3.6 and the pypy3 binary works great. To build pypy myself, per the pypy documentation, I have taken the following steps: $ hg clone http://foss.heptapod.net/pypy/pypy pypy $ cd pypy $ hg up py3.6 $ cd pypy/goal $ pypy3 ../../rpython/bin/rpython --opt=jit This leads to a series of SyntaxErrors related to print. It appears that the build-related files in the branch are still intended for pypy version < 3. I am probably missing something obvious and would appreciate your advice. Thanks. -gyro -------------- next part -------------- A non-text attachment was scrubbed... Name: pEpkey.asc Type: application/pgp-keys Size: 1765 bytes Desc: not available URL: From gyromagnetic at gmail.com Tue Sep 1 18:05:02 2020 From: gyromagnetic at gmail.com (gyro funch) Date: Tue, 01 Sep 2020 22:05:02 -0000 Subject: [pypy-dev] We have officially moved to foss.heptapod.net/pypy In-Reply-To: <8bfdeade-5949-e46b-494b-a3c6721b38e3@gmail.com> References: <8bfdeade-5949-e46b-494b-a3c6721b38e3@gmail.com> Message-ID: The new website looks great! -gyro On 2/16/2020 11:12 AM, Matti Picus wrote: > As reported in the recent blog post > https://morepypy.blogspot.com/2020/02/pypy-and-cffi-have-moved-to-heptapod.html, > please do not add content to bitbucket.org/pypy or any of the repos > there. From now on use https://foss.heptapod.net/pypy/pypy. The repos > will continue to live on bitbucket until May 31, and we are still > hosting our wiki and downloads there, but activity should move to the > new instance. Anyone can open an issue, but in order to commit (as > explained in the previous mail) you will need permissions on the repo. > The FOSS heptapod instance does not support personal forks, so for now > PRs will be branches (heptapod recommends topic branches but some of us > are not convinced) on the main repo. > > > Also take a look at the facelift on www.pypy.org. > > > Matti From matt at vazor.com Tue Sep 1 18:24:06 2020 From: matt at vazor.com (Matt Billenstein) Date: Tue, 1 Sep 2020 22:24:06 +0000 Subject: [pypy-dev] help building pypy In-Reply-To: References: Message-ID: <010101744bc526e0-c2c3775e-cbe6-47d3-aaf6-58f8a0e28699-000000@us-west-2.amazonses.com> On Tue, Sep 01, 2020 at 03:58:13PM -0600, gyro funch wrote: > Hi, > > I am interested in building pypy 3.6 on Ubuntu 20.04 LTS. > > ... > $ pypy3 ../../rpython/bin/rpython --opt=jit > > This leads to a series of SyntaxErrors related to print. It appears that > the build-related files in the branch are still intended for pypy > version < 3. You need pypy2 to build pypy. m -- Matt Billenstein matt at vazor.com http://www.vazor.com/ From matti.picus at gmail.com Tue Sep 8 16:15:47 2020 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 8 Sep 2020 23:15:47 +0300 Subject: [pypy-dev] time for a new release Message-ID: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> An HTML attachment was scrubbed... URL: From niklas.bivald at enplore.com Tue Sep 8 17:12:43 2020 From: niklas.bivald at enplore.com (Niklas B) Date: Tue, 8 Sep 2020 23:12:43 +0200 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ Message-ID: Hi, I?ve been trying to build data science library pyarrow (the arrow library, for parquet files mainly in my case) for PyPy. I?ve gotten it working for pypy2 a few years ago, and is now trying for PyPy3. Overall I get it to build and produce a pyarrow wheel file by following the arrow instructions. So far so good. I expect a massive part of pyarrow not to work, but for my case I really only need `pandas.read_parquet`. However I am stuck trying to figure out how to use the pyppy cpyext for Datetime. The code I?m trying to build is: https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datetime.cc#L37 (older branch, better luck in building it) Which is basically: PyDateTime_CAPI* datetime_api = nullptr; void InitDatetime() { PyAcquireGIL lock; datetime_api = reinterpret_cast(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); if (datetime_api == nullptr) { Py_FatalError("Could not import datetime C API"); } } I?ve tried about a million different ways, but I?m way outside my comfort zone :) I can get it to build by doing: datetime_api = PyDateTimeAPI; And also: datetime_api = reinterpret_cast(PyCapsule_Import("datetime", 0)); And: datetime_api = reinterpret_cast(PyCapsule_Import("datetime.datetime_CAPI", 0)); But both of these trigger the fatal error in the code after (?could not import date time C API? or "PyCapsule_Import "datetime" is not valid? or module 'datetime' has no attribute 'datetime_CAPI') I will be posting reproducible builds once I get them working. I am more than happy to pay 300USD to anyone (or to PyPy) who can help me getting this to run: Import pandas d = pandas.read_parquet(?file.parq?) Obviously that?s not enough money to cover things but at least it?s something :) obviously all results and builds will be public Regards, Niklas -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Sep 8 18:06:24 2020 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 9 Sep 2020 01:06:24 +0300 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: Message-ID: Have you tried the documented interface?https://docs.python.org/3.6/c-api/datetime.html which is to dispense with all that code and use #import |PyDateTime_IMPORT| which does the right thing on each implementation (CPython - does the PyCapsule_Import; PyPy - calls _PyDateTime_Import() ) A donation to https://opencollective.com/pypy is always appreciated. Matti On 9/9/20 12:12 AM, Niklas B wrote: > Hi, > > I?ve been trying to build data science library pyarrow (the arrow > library, for parquet files mainly in my case) for PyPy. I?ve gotten it > working for pypy2 a few years ago, and is now trying for PyPy3. > Overall I get it to build and produce a pyarrow wheel file by > following the arrow instructions. So far so good. I expect a massive > part of pyarrow not to work, but for my case I really only need > `pandas.read_parquet`. However I am stuck trying to figure out how to > use the pyppy cpyext for Datetime. > > The code I?m trying to build is: > > https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datetime.cc#L37?(older > branch, better luck in building it) > > Which is basically: > > PyDateTime_CAPI* datetime_api = nullptr; > > > > void InitDatetime() { > > PyAcquireGIL lock; > > datetime_api = > > reinterpret_cast(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); > > if (datetime_api == nullptr) { > > Py_FatalError("Could not import datetime C API"); > > } > > } > > > I?ve tried about a million different ways, but I?m way outside my > comfort zone :) I can get it to build by doing: > > datetime_api = PyDateTimeAPI; > > And also: > > datetime_api = > reinterpret_cast(PyCapsule_Import("datetime", 0)); > > And: > > datetime_api = > reinterpret_cast(PyCapsule_Import("datetime.datetime_CAPI", > 0)); > > But both of these trigger the fatal error in the code after (?could > not import date time C API? or "PyCapsule_Import "datetime" is not > valid? or module 'datetime' has no attribute 'datetime_CAPI') > > I will be posting reproducible builds once I get them working. > > I am more than happy to pay 300USD to anyone (or to PyPy) who can help > me getting this to run: > > Import pandas > d = pandas.read_parquet(?file.parq?) > > Obviously that?s not enough money to cover things but at least it?s > something :) obviously all results and builds will be public > > Regards, > Niklas > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From mgorny at gentoo.org Wed Sep 9 02:55:19 2020 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Wed, 09 Sep 2020 08:55:19 +0200 Subject: [pypy-dev] time for a new release In-Reply-To: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> Message-ID: On Tue, 2020-09-08 at 23:15 +0300, Matti Picus wrote: > I have uploaded rc1 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) which should be mirrored soon to https://downloads.python.org/pypy/ > > The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download_advanced.rst#L465 > > The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html > > This release does include a 3.7 alpha. > > Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them. > > Any comments are welcome. > What's the vulnerability status of stdlib? I've tested pypy2.7 and pypy3.6 so far and neither seems to contain CVE- 2019-20907 fix (it was never backported to py2.7), the patch from [1] seems to apply cleanly to both. pypy3.6 seems to be missing bpo-39603, and the patch from [2] doesn't apply cleanly (does pypy3 contain outdated version or modified?). CVE-2020-14422 is also unresolved. Could you please either update stdlib of pypy3.6 or look through CPython changes and backport the security fixes? For pypy2.7, please backport [1] directly since upstream is no longer maintaining that branch. [1] https://github.com/python/cpython/commit/47a2955589bdb1a114d271496ff803ad73f954b8 [2] https://github.com/python/cpython/commit/f02de961b9f19a5db0ead56305fe0057a78787ae -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Wed Sep 9 05:41:00 2020 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 9 Sep 2020 12:41:00 +0300 Subject: [pypy-dev] time for a new release In-Reply-To: References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> Message-ID: <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> On 9/9/20 9:55 AM, Micha? G?rny wrote: > On Tue, 2020-09-08 at 23:15 +0300, Matti Picus wrote: >> I have uploaded rc1 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) which should be mirrored soon to https://downloads.python.org/pypy/ >> >> The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download_advanced.rst#L465 >> >> The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html >> >> This release does include a 3.7 alpha. >> >> Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them. >> >> Any comments are welcome. >> > What's the vulnerability status of stdlib? > > I've tested pypy2.7 and pypy3.6 so far and neither seems to contain CVE- > 2019-20907 fix (it was never backported to py2.7), the patch from [1] > seems to apply cleanly to both. > > pypy3.6 seems to be missing bpo-39603, and the patch from [2] doesn't > apply cleanly (does pypy3 contain outdated version or modified?). > > CVE-2020-14422 is also unresolved. > > Could you please either update stdlib of pypy3.6 or look through CPython > changes and backport the security fixes? For pypy2.7, please backport > [1] directly since upstream is no longer maintaining that branch. > > [1] https://github.com/python/cpython/commit/47a2955589bdb1a114d271496ff803ad73f954b8 > [2] https://github.com/python/cpython/commit/f02de961b9f19a5db0ead56305fe0057a78787ae > Thanks for looking at this.We ship stdlib 2.7.13, 3.6.9, 3.7.4 with some slight modifications, including backporting some fixes. I fixed CVE-2019-20907 for? pypy2.7, pypy3,6, and CVE-2020-14422 for py3.6, 3.7 bpo-39603 is part of 3.6.12, 3.7.9 which were shipped 25 days ago, and that file has changed significantly since the versions we ship. Updating the stdlib is a large undertaking, help welcome for py2.7 and py3.7. I don't think it is worth the effort for py3.6. There are directions in lib-python/stdlib-upgrade.txt. Matti From niklas.bivald at enplore.com Wed Sep 9 08:08:15 2020 From: niklas.bivald at enplore.com (Niklas B) Date: Wed, 9 Sep 2020 14:08:15 +0200 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: Message-ID: Thank you, I managed to get it built using an older version (instructions and whl file for those who are interested is available at https://github.com/bivald/pyarrow-on-pypy3 ) I have setup a monthly 50USD recurring donation (under ?Enplore?) > On 9 Sep 2020, at 00:06, Matti Picus wrote: > > Have you tried the documented interface https://docs.python.org/3.6/c-api/datetime.html > > which is to dispense with all that code and use > > > #import > > |PyDateTime_IMPORT| > > > which does the right thing on each implementation (CPython - does the PyCapsule_Import; PyPy - calls _PyDateTime_Import() ) > > > A donation to https://opencollective.com/pypy is always appreciated. > > Matti > > > On 9/9/20 12:12 AM, Niklas B wrote: >> Hi, >> >> I?ve been trying to build data science library pyarrow (the arrow library, for parquet files mainly in my case) for PyPy. I?ve gotten it working for pypy2 a few years ago, and is now trying for PyPy3. Overall I get it to build and produce a pyarrow wheel file by following the arrow instructions. So far so good. I expect a massive part of pyarrow not to work, but for my case I really only need `pandas.read_parquet`. However I am stuck trying to figure out how to use the pyppy cpyext for Datetime. >> >> The code I?m trying to build is: >> >> https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datetime.cc#L37 (older branch, better luck in building it) >> >> Which is basically: >> >> PyDateTime_CAPI* datetime_api = nullptr; >> >> >> >> void InitDatetime() { >> >> PyAcquireGIL lock; >> >> datetime_api = >> >> reinterpret_cast(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); >> >> if (datetime_api == nullptr) { >> >> Py_FatalError("Could not import datetime C API"); >> >> } >> >> } >> >> >> I?ve tried about a million different ways, but I?m way outside my comfort zone :) I can get it to build by doing: >> >> datetime_api = PyDateTimeAPI; >> >> And also: >> >> datetime_api = reinterpret_cast(PyCapsule_Import("datetime", 0)); >> >> And: >> >> datetime_api = reinterpret_cast(PyCapsule_Import("datetime.datetime_CAPI", 0)); >> >> But both of these trigger the fatal error in the code after (?could not import date time C API? or "PyCapsule_Import "datetime" is not valid? or module 'datetime' has no attribute 'datetime_CAPI') >> >> I will be posting reproducible builds once I get them working. >> >> I am more than happy to pay 300USD to anyone (or to PyPy) who can help me getting this to run: >> >> Import pandas >> d = pandas.read_parquet(?file.parq?) >> >> Obviously that?s not enough money to cover things but at least it?s something :) obviously all results and builds will be public >> >> Regards, >> Niklas >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Sep 9 12:07:28 2020 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 9 Sep 2020 19:07:28 +0300 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: Message-ID: <871e29b5-7395-d53e-de66-2055681b8358@gmail.com> On 9/9/20 3:08 PM, Niklas B wrote: > Thank you, I managed to get it built using an older version > (instructions and whl file for those who are interested is available > at https://github.com/bivald/pyarrow-on-pypy3) > > Cool. It would be nice to backport the datetime changes, I think you could put in a #ifdef PYPY_VERSION macro and add the code I pasted. I'll se what I can do about PyMemoryView_GetContiguous. Are there more missing C-API functions? Thanks for the donation(s)! Matti From mgorny at gentoo.org Wed Sep 9 17:04:55 2020 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Wed, 09 Sep 2020 23:04:55 +0200 Subject: [pypy-dev] time for a new release In-Reply-To: <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> Message-ID: On Wed, 2020-09-09 at 12:41 +0300, Matti Picus wrote: > On 9/9/20 9:55 AM, Micha? G?rny wrote: > > On Tue, 2020-09-08 at 23:15 +0300, Matti Picus wrote: > > > I have uploaded rc1 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) which should be mirrored soon to https://downloads.python.org/pypy/ > > > > > > The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download_advanced.rst#L465 > > > > > > The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html > > > > > > This release does include a 3.7 alpha. > > > > > > Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them. > > > > > > Any comments are welcome. > > > > > What's the vulnerability status of stdlib? > > > > I've tested pypy2.7 and pypy3.6 so far and neither seems to contain CVE- > > 2019-20907 fix (it was never backported to py2.7), the patch from [1] > > seems to apply cleanly to both. > > > > pypy3.6 seems to be missing bpo-39603, and the patch from [2] doesn't > > apply cleanly (does pypy3 contain outdated version or modified?). > > > > CVE-2020-14422 is also unresolved. > > > > Could you please either update stdlib of pypy3.6 or look through CPython > > changes and backport the security fixes? For pypy2.7, please backport > > [1] directly since upstream is no longer maintaining that branch. > > > > [1] https://github.com/python/cpython/commit/47a2955589bdb1a114d271496ff803ad73f954b8 > > [2] https://github.com/python/cpython/commit/f02de961b9f19a5db0ead56305fe0057a78787ae > > > > Thanks for looking at this.We ship stdlib 2.7.13, 3.6.9, 3.7.4 with some > slight modifications, including backporting some fixes. > > > I fixed CVE-2019-20907 for pypy2.7, pypy3,6, and CVE-2020-14422 for > py3.6, 3.7 > > bpo-39603 is part of 3.6.12, 3.7.9 which were shipped 25 days ago, and that file has changed significantly since the versions we ship. > > Updating the stdlib is a large undertaking, help welcome for py2.7 and > py3.7. I don't think it is worth the effort for py3.6. Is this something to be done just before the release? Would you accept fixes rebased specifically on top of current pypy code? Unless I'm mistaken, bpo-39603 should be trivial to fix. I can submit a merge request if you want. However, it's so trivial it'd probably take you less time to fix it yourself than me to recall how to use mercurial again ;-). -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Wed Sep 9 17:41:54 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 10 Sep 2020 00:41:54 +0300 Subject: [pypy-dev] time for a new release In-Reply-To: References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> Message-ID: <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> On 9/10/20 12:04 AM, Micha? G?rny wrote: > Is this something to be done just before the release? Would you accept > fixes rebased specifically on top of current pypy code? > > Unless I'm mistaken, bpo-39603 should be trivial to fix. I can submit > a merge request if you want. I am willing to accept small patches to stdlib in the next two days (unless something delays the release more). I don't have time to work on bpo-39603. but am willing to review a MR or even patch diff via email. Matti From matti.picus at gmail.com Thu Sep 10 05:30:05 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 10 Sep 2020 12:30:05 +0300 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From mgorny at gentoo.org Thu Sep 10 05:40:32 2020 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Thu, 10 Sep 2020 11:40:32 +0200 Subject: [pypy-dev] time for a new release In-Reply-To: <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> Message-ID: <7c2164789bd63268edc06dafa1b1a23f3a5f2e73.camel@gentoo.org> On Thu, 2020-09-10 at 00:41 +0300, Matti Picus wrote: > On 9/10/20 12:04 AM, Micha? G?rny wrote: > > > Is this something to be done just before the release? Would you accept > > fixes rebased specifically on top of current pypy code? > > > > Unless I'm mistaken, bpo-39603 should be trivial to fix. I can submit > > a merge request if you want. > > I am willing to accept small patches to stdlib in the next two days > (unless something delays the release more). I don't have time to work on > bpo-39603. but am willing to review a MR or even patch diff via email. > I've filed two MRs for a start: https://foss.heptapod.net/pypy/pypy/-/merge_requests/752 https://foss.heptapod.net/pypy/pypy/-/merge_requests/753 Could you please let me know if that approach is ok? If so, I'll tackle the remaining vulnerabilities (around 6 patches FWICS). -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Thu Sep 10 06:37:29 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 10 Sep 2020 13:37:29 +0300 Subject: [pypy-dev] time for a new release In-Reply-To: <7c2164789bd63268edc06dafa1b1a23f3a5f2e73.camel@gentoo.org> References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> <7c2164789bd63268edc06dafa1b1a23f3a5f2e73.camel@gentoo.org> Message-ID: <17db3ec0-0279-8255-8e8c-48b3c497221c@gmail.com> On 9/10/20 12:40 PM, Micha? G?rny wrote: > I've filed two MRs for a start: > https://foss.heptapod.net/pypy/pypy/-/merge_requests/752 > https://foss.heptapod.net/pypy/pypy/-/merge_requests/753 > > Could you please let me know if that approach is ok? If so, I'll tackle > the remaining vulnerabilities (around 6 patches FWICS). > Thanks. Could you add comments in each patched chunk of each PR that will allow us to track how the code diverged? Something like "added manually to resolve bpo xxx". I am concerned this will make an eventual update to the stdlib difficult. MR753 is quite invasive. Are all the other patches against the same single file? Maybe a different strategy would be to adopt the file and tests as-is from the HEAD of the CPython branch, as long as no user-facing APIs change. It would be nice to make similar PRs against default (python2.7) in lib-python/2.7 on that branch. Matti From mgorny at gentoo.org Thu Sep 10 06:45:44 2020 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Thu, 10 Sep 2020 12:45:44 +0200 Subject: [pypy-dev] time for a new release In-Reply-To: <17db3ec0-0279-8255-8e8c-48b3c497221c@gmail.com> References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> <7c2164789bd63268edc06dafa1b1a23f3a5f2e73.camel@gentoo.org> <17db3ec0-0279-8255-8e8c-48b3c497221c@gmail.com> Message-ID: <43674b0ce74d9b5747065bb61e58ba24bbddd23b.camel@gentoo.org> On Thu, 2020-09-10 at 13:37 +0300, Matti Picus wrote: > On 9/10/20 12:40 PM, Micha? G?rny wrote: > > I've filed two MRs for a start: > > https://foss.heptapod.net/pypy/pypy/-/merge_requests/752 > > https://foss.heptapod.net/pypy/pypy/-/merge_requests/753 > > > > Could you please let me know if that approach is ok? If so, I'll tackle > > the remaining vulnerabilities (around 6 patches FWICS). > > > > Thanks. > > Could you add comments in each patched chunk of each PR that will allow > us to track how the code diverged? Something like "added manually to > resolve bpo xxx". I don't see a problem with doing that. > > > I am concerned this will make an eventual update to the stdlib > difficult. MR753 is quite invasive. Are all the other patches against > the same single file? Maybe a different strategy would be to adopt the > file and tests as-is from the HEAD of the CPython branch, as long as no > user-facing APIs change. I suppose I can try doing that. I presumed you'll want to keep the changes to bare minimum possible but it might turn out not much different once I iterate over all the CVEs. > It would be nice to make similar PRs against default (python2.7) in > lib-python/2.7 on that branch. The primary problem with that is that CPython 2.7 is abandoned and contrary to all the big talk about random people willing to maintain it post-EOL, I am yet to see anyone taking its security seriously. So far I and the Fedora maintainer were able to independently backport one vulnerability that clearly applied (the tarfile one) but we weren't able to get a clear match of any other Python 3.x fixes to 2.7 codebase. Well, until today when thanks to you I've noticed that http.request code has a vulnerable match in httplib. But this all is lots of work, and I'm really supposed to be doing something else right now. I'm trying my best but I'm not sure if I can manage to fix several months of negligence in two days. -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Thu Sep 10 07:00:03 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 10 Sep 2020 14:00:03 +0300 Subject: [pypy-dev] time for a new release In-Reply-To: <43674b0ce74d9b5747065bb61e58ba24bbddd23b.camel@gentoo.org> References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> <7c2164789bd63268edc06dafa1b1a23f3a5f2e73.camel@gentoo.org> <17db3ec0-0279-8255-8e8c-48b3c497221c@gmail.com> <43674b0ce74d9b5747065bb61e58ba24bbddd23b.camel@gentoo.org> Message-ID: <841fcb2d-e4df-6dfd-d2e6-20d97aed18fa@gmail.com> On 9/10/20 1:45 PM, Micha? G?rny wrote: > So far I and the Fedora maintainer were able to independently backport > one vulnerability that clearly applied (the tarfile one) but we weren't > able to get a clear match of any other Python 3.x fixes to 2.7 codebase. > Well, until today when thanks to you I've noticed that http.request > code has a vulnerable match in httplib. > > But this all is lots of work, and I'm really supposed to be doing > something else right now. I'm trying my best but I'm not sure if I can > manage to fix several months of negligence in two days. Thanks for all you are doing. The release deadline is only a motivator for now since we could do another much smaller release next month if needed. I want to move toward python3.7 as soon as possible since the scientific python stack's stated python version policy means 3.6 will no longer be expressly supported especially after 3.9 comes out. Matti From mgorny at gentoo.org Thu Sep 10 07:08:34 2020 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Thu, 10 Sep 2020 13:08:34 +0200 Subject: [pypy-dev] time for a new release In-Reply-To: <841fcb2d-e4df-6dfd-d2e6-20d97aed18fa@gmail.com> References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> <7c2164789bd63268edc06dafa1b1a23f3a5f2e73.camel@gentoo.org> <17db3ec0-0279-8255-8e8c-48b3c497221c@gmail.com> <43674b0ce74d9b5747065bb61e58ba24bbddd23b.camel@gentoo.org> <841fcb2d-e4df-6dfd-d2e6-20d97aed18fa@gmail.com> Message-ID: On Thu, 2020-09-10 at 14:00 +0300, Matti Picus wrote: > On 9/10/20 1:45 PM, Micha? G?rny wrote: > > So far I and the Fedora maintainer were able to independently backport > > one vulnerability that clearly applied (the tarfile one) but we weren't > > able to get a clear match of any other Python 3.x fixes to 2.7 codebase. > > Well, until today when thanks to you I've noticed that http.request > > code has a vulnerable match in httplib. > > > > But this all is lots of work, and I'm really supposed to be doing > > something else right now. I'm trying my best but I'm not sure if I can > > manage to fix several months of negligence in two days. > > Thanks for all you are doing. The release deadline is only a motivator > for now since we could do another much smaller release next month if needed. > > I want to move toward python3.7 as soon as possible since the scientific > python stack's stated python version policy means 3.6 will no longer be > expressly supported especially after 3.9 comes out. > I wholeheartedly support that. Gentoo has switched to Python 3.7 by default already, and it sucks that the only version of PyPy3 is behind that. That said, have you considered going straight to 3.8? In my experience, the switch from 3.7 to 3.8 was far less painful than from 3.6 to 3.7, so it might be less work to skip 3.7 entirely. -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From niklas.bivald at enplore.com Thu Sep 10 07:16:59 2020 From: niklas.bivald at enplore.com (Niklas B) Date: Thu, 10 Sep 2020 13:16:59 +0200 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: Message-ID: <1ECAEE19-B13B-4403-9E8F-1ACDCFF2E113@enplore.com> I will try, I tried with the 2020-09-10 (https://buildbot.pypy.org/nightly/py3.6/pypy-c-jit-latest-linux64.tar.bz2 ) but your changes might have not made it into that one. The errors I get is: https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python/common.h#L256 In file included from /arrow/cpp/src/arrow/python/common.cc:18: /arrow/cpp/src/arrow/python/common.h: In member function ?arrow::Status arrow::py::PyBytesView::FromBinary(PyObject*, const char*)?: /arrow/cpp/src/arrow/python/common.h:256:31: error: ?PyMemoryView_GetContiguous? was not declared in this scope PyObject* contig_view = PyMemoryView_GetContiguous(obj, PyBUF_READ, 'C'); ^~~~~~~~~~~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/common.h:256:31: note: suggested alternative: ?PyMemoryView_FromMemory? PyObject* contig_view = PyMemoryView_GetContiguous(obj, PyBUF_READ, 'C'); ^~~~~~~~~~~~~~~~~~~~~~~~~~ PyMemoryView_FromMemory Or I might need to patch something :) Odd thing is that running it on the latest code I don?t get any errors on date time, only on above. That said, it might simply be that the errors are ?below? the above errors I did a new branch for latest: https://github.com/bivald/pyarrow-on-pypy3/blob/feature/latest-pypy-latest-pyarrow/Dockerfile > On 10 Sep 2020, at 11:30, Matti Picus wrote: > > I implemented the easy part (without allocating and copying non-contiguous data to a new contiguous buffer) of PyMemoryView_GetContiguous, which will make it into the upcoming release. You can try it out from tonight's nightlies > Matti > > On 9/9/20 3:08 PM, Niklas B wrote: >> Thank you, I managed to get it built using an older version (instructions and whl file for those who are interested is available at https://github.com/bivald/pyarrow-on-pypy3 ) >> >> I have setup a monthly 50USD recurring donation (under ?Enplore?) >> >>> On 9 Sep 2020, at 00:06, Matti Picus > wrote: >>> >>> Have you tried the documented interface https://docs.python.org/3.6/c-api/datetime.html >>> >>> which is to dispense with all that code and use >>> >>> >>> #import >>> >>> |PyDateTime_IMPORT| >>> >>> >>> which does the right thing on each implementation (CPython - does the PyCapsule_Import; PyPy - calls _PyDateTime_Import() ) >>> >>> >>> A donation to https://opencollective.com/pypy is always appreciated. >>> >>> Matti >>> >>> >>> On 9/9/20 12:12 AM, Niklas B wrote: >>>> Hi, >>>> >>>> I?ve been trying to build data science library pyarrow (the arrow library, for parquet files mainly in my case) for PyPy. I?ve gotten it working for pypy2 a few years ago, and is now trying for PyPy3. Overall I get it to build and produce a pyarrow wheel file by following the arrow instructions. So far so good. I expect a massive part of pyarrow not to work, but for my case I really only need `pandas.read_parquet`. However I am stuck trying to figure out how to use the pyppy cpyext for Datetime. >>>> >>>> The code I?m trying to build is: >>>> >>>> https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/python/datetime.cc#L37 (older branch, better luck in building it) >>>> >>>> Which is basically: >>>> >>>> PyDateTime_CAPI* datetime_api = nullptr; >>>> >>>> >>>> >>>> void InitDatetime() { >>>> >>>> PyAcquireGIL lock; >>>> >>>> datetime_api = >>>> >>>> reinterpret_cast(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); >>>> >>>> if (datetime_api == nullptr) { >>>> >>>> Py_FatalError("Could not import datetime C API"); >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> I?ve tried about a million different ways, but I?m way outside my comfort zone :) I can get it to build by doing: >>>> >>>> datetime_api = PyDateTimeAPI; >>>> >>>> And also: >>>> >>>> datetime_api = reinterpret_cast(PyCapsule_Import("datetime", 0)); >>>> >>>> And: >>>> >>>> datetime_api = reinterpret_cast(PyCapsule_Import("datetime.datetime_CAPI", 0)); >>>> >>>> But both of these trigger the fatal error in the code after (?could not import date time C API? or "PyCapsule_Import "datetime" is not valid? or module 'datetime' has no attribute 'datetime_CAPI') >>>> >>>> I will be posting reproducible builds once I get them working. >>>> >>>> I am more than happy to pay 300USD to anyone (or to PyPy) who can help me getting this to run: >>>> >>>> Import pandas >>>> d = pandas.read_parquet(?file.parq?) >>>> >>>> Obviously that?s not enough money to cover things but at least it?s something :) obviously all results and builds will be public >>>> >>>> Regards, >>>> Niklas >>>> >>>> _______________________________________________ >>>> pypy-dev mailing list >>>> pypy-dev at python.org >>>> https://mail.python.org/mailman/listinfo/pypy-dev >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Thu Sep 10 07:27:12 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 10 Sep 2020 14:27:12 +0300 Subject: [pypy-dev] time for a new release In-Reply-To: References: <19a60020-0d2d-ac11-996f-6aab119823b0@gmail.com> <36760ced-1b1f-c48e-79f1-266c9c0ecb5f@gmail.com> <26826cce-117f-bced-a9cc-bb97985b6cfd@gmail.com> <7c2164789bd63268edc06dafa1b1a23f3a5f2e73.camel@gentoo.org> <17db3ec0-0279-8255-8e8c-48b3c497221c@gmail.com> <43674b0ce74d9b5747065bb61e58ba24bbddd23b.camel@gentoo.org> <841fcb2d-e4df-6dfd-d2e6-20d97aed18fa@gmail.com> Message-ID: On 9/10/20 2:08 PM, Micha? G?rny wrote: > I wholeheartedly support that. Gentoo has switched to Python 3.7 by > default already, and it sucks that the only version of PyPy3 is behind > that. That said, have you considered going straight to 3.8? In my > experience, the switch from 3.7 to 3.8 was far less painful than from > 3.6 to 3.7, so it might be less work to skip 3.7 entirely. We are on the cusp of releasing 3.7, I don't want to delay any more. Hopefully we will have some bandwidth to push forward soon. It is encouraging to hear that the transition may be easier than previous ones. Matti From matti.picus at gmail.com Thu Sep 10 07:28:58 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 10 Sep 2020 14:28:58 +0300 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: <1ECAEE19-B13B-4403-9E8F-1ACDCFF2E113@enplore.com> References: <1ECAEE19-B13B-4403-9E8F-1ACDCFF2E113@enplore.com> Message-ID: On 9/10/20 2:16 PM, Niklas B wrote: > I will try, I tried with the 2020-09-10 > (https://buildbot.pypy.org/nightly/py3.6/pypy-c-jit-latest-linux64.tar.bz2) > but your changes might have not made it into that one. The errors I > get is: > > Sorry. I meant from *tomorrow's* nightly, not from todays, I just commited the change so it will appear in the runs in ~16 hours. Matti From niklas.bivald at enplore.com Thu Sep 10 07:30:10 2020 From: niklas.bivald at enplore.com (Niklas B) Date: Thu, 10 Sep 2020 13:30:10 +0200 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: <1ECAEE19-B13B-4403-9E8F-1ACDCFF2E113@enplore.com> Message-ID: <88054D70-885B-4E03-8F09-FE4435CFCAC8@enplore.com> That?s what I figured, cool, I will try it! > On 10 Sep 2020, at 13:28, Matti Picus wrote: > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From niklas.bivald at enplore.com Fri Sep 11 06:01:11 2020 From: niklas.bivald at enplore.com (Niklas B) Date: Fri, 11 Sep 2020 12:01:11 +0200 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: <88054D70-885B-4E03-8F09-FE4435CFCAC8@enplore.com> References: <1ECAEE19-B13B-4403-9E8F-1ACDCFF2E113@enplore.com> <88054D70-885B-4E03-8F09-FE4435CFCAC8@enplore.com> Message-ID: The PyMemoryView_GetContiguous errors are all gone - good work! It didn?t really like my butchering of datetime.cc https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python/datetime.cc#L37 Added: #include And PyDateTime_IMPORT Then changed: datetime_api = reinterpret_cast(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); To: datetime_api = PyDateTimeAPI; They do some PyDateTimeAPI voodoo at https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python/datetime.h#L29 which might be a culprit as well. Gives the following outputs: /arrow/cpp/src/arrow/python/datetime.h:34: warning: "PyDateTimeAPI" redefined #define PyDateTimeAPI ::arrow::py::internal::datetime_api In file included from /opt/pypy/include/Python.h:144, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/pypy_decl.h:1121: note: this is the location of the previous definition #define PyDateTimeAPI PyPyDateTimeAPI In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:4:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? PyTypeObject *DateType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:5:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? PyTypeObject *DateTimeType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:6:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? PyTypeObject *TimeType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:7:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? PyTypeObject *DeltaType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:8:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? PyTypeObject *TZInfoType; ^~~~~~~~~~~~ PyType_Check /opt/pypy/include/cpyext_datetime.h:11:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *(*Date_FromDate)(int, int, int, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:12:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *(*DateTime_FromDateAndTime)(int, int, int, int, int, int, int, ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:14:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *(*Time_FromTime)(int, int, int, int, PyObject*, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:15:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *(*Delta_FromDelta)(int, int, int, int, PyTypeObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:18:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *(*DateTime_FromTimestamp)(PyObject*, PyObject*, PyObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:19:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *(*Date_FromTimestamp)(PyObject*, PyObject*); ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:24:5: error: ?PyObject_HEAD? does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:24:5: note: the macro ?PyObject_HEAD? had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:35:5: error: ?PyObject_HEAD? does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:35:5: note: the macro ?PyObject_HEAD? had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:37:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *tzinfo; ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:42:5: error: ?PyObject_HEAD? does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:42:5: note: the macro ?PyObject_HEAD? had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:44:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? PyObject *tzinfo; ^~~~~~~~ PyObject_New /opt/pypy/include/cpyext_datetime.h:49:5: error: ?PyObject_HEAD? does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:49:5: note: the macro ?PyObject_HEAD? had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /opt/pypy/include/datetime.h:7, from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/cpyext_datetime.h:54:5: error: ?PyObject_HEAD? does not name a type PyObject_HEAD ^~~~~~~~~~~~~ /opt/pypy/include/cpyext_datetime.h:54:5: note: the macro ?PyObject_HEAD? had not yet been defined In file included from /opt/pypy/include/object.h:10, from /opt/pypy/include/Python.h:79, from /arrow/cpp/src/arrow/python/platform.h:23, from /arrow/cpp/src/arrow/python/pyarrow.h:20, from /arrow/cpp/src/arrow/python/common.h:24, from /arrow/cpp/src/arrow/python/datetime.cc:24: /opt/pypy/include/cpyext_object.h:5: note: it was later defined here #define PyObject_HEAD \ In file included from /arrow/cpp/src/arrow/python/datetime.cc:22: /opt/pypy/include/datetime.h:9:30: error: expected constructor, destructor, or type conversion before ?PyDateTimeAPI? PyAPI_DATA(PyDateTime_CAPI*) PyDateTimeAPI; ^~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:37:1: error: expected ?)? before ?=? token PyDateTime_IMPORT ^~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ?arrow::Status arrow::py::internal::PyTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)?: /arrow/cpp/src/arrow/python/datetime.cc:237:10: error: ?struct PyDateTime_CAPI? has no member named ?Time_FromTime? *out = PyTime_FromTime(static_cast(hour), static_cast(minute), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:237:10: error: ?struct PyDateTime_CAPI? has no member named ?TimeType? *out = PyTime_FromTime(static_cast(hour), static_cast(minute), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ?arrow::Status arrow::py::internal::PyDate_from_int(int64_t, arrow::DateUnit, PyObject**)?: /arrow/cpp/src/arrow/python/datetime.cc:245:10: error: ?struct PyDateTime_CAPI? has no member named ?Date_FromDate? *out = PyDate_FromDate(static_cast(year), static_cast(month), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:245:10: error: ?struct PyDateTime_CAPI? has no member named ?DateType? *out = PyDate_FromDate(static_cast(year), static_cast(month), ^~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc: In function ?arrow::Status arrow::py::internal::PyDateTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)?: /arrow/cpp/src/arrow/python/datetime.cc:257:10: error: ?struct PyDateTime_CAPI? has no member named ?DateTime_FromDateAndTime? *out = PyDateTime_FromDateAndTime( ^~~~~~~~~~~~~~~~~~~~~~~~~~ /arrow/cpp/src/arrow/python/datetime.cc:257:10: error: ?struct PyDateTime_CAPI? has no member named ?DateTimeType? *out = PyDateTime_FromDateAndTime( ^~~~~~~~~~~~~~~~~~~~~~~~~~ make[2]: *** [src/arrow/python/CMakeFiles/arrow_python_objlib.dir/build.make:121: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/datetime.cc.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:1628: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/all] Error > On 10 Sep 2020, at 13:30, Niklas B wrote: > > That?s what I figured, cool, I will try it! > >> On 10 Sep 2020, at 13:28, Matti Picus > wrote: >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Fri Sep 11 08:18:36 2020 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 11 Sep 2020 15:18:36 +0300 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: <1ECAEE19-B13B-4403-9E8F-1ACDCFF2E113@enplore.com> <88054D70-885B-4E03-8F09-FE4435CFCAC8@enplore.com> Message-ID: Thse changes seems to compile for me, but I did not run the tests. https://gist.github.com/mattip/c9c8398b58721ae5893dc8134c353f28 Matti On 9/11/20 1:01 PM, Niklas B wrote: > The?PyMemoryView_GetContiguous errors are all gone - good work! > > It didn?t really like my butchering of datetime.cc > > https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python/datetime.cc#L37 > > Added: > #include > > And > PyDateTime_IMPORT > > > > Then changed: > datetime_api = > > reinterpret_cast(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); > > > To: > > ? datetime_api = PyDateTimeAPI; > > They do some PyDateTimeAPI voodoo at > https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python/datetime.h#L29?which > might be a culprit as well. > > > > > > > > > > Gives the following outputs: > /arrow/cpp/src/arrow/python/datetime.h:34: warning: "PyDateTimeAPI" > redefined > ?#define PyDateTimeAPI ::arrow::py::internal::datetime_api > > In file included from /opt/pypy/include/Python.h:144, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/platform.h:23, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/pyarrow.h:20, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/common.h:24, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:24 > : > /opt/pypy/include/pypy_decl.h:1121: note: this is the location of the > previous definition > ?#define PyDateTimeAPI PyPyDateTimeAPI > > In file included from /opt/pypy/include/datetime.h:7, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:22 > : > /opt/pypy/include/cpyext_datetime.h:4:5: error: ?PyTypeObject? does > not name a type; did you mean ?PyType_Check?? > ? ? ?PyTypeObject *DateType; > ? ? ?^~~~~~~~~~~~ > ? ? ?PyType_Check > /opt/pypy/include/cpyext_datetime.h:5:5: error: ?PyTypeObject? does > not name a type; did you mean ?PyType_Check?? > ? ? ?PyTypeObject *DateTimeType; > ? ? ?^~~~~~~~~~~~ > ? ? ?PyType_Check > /opt/pypy/include/cpyext_datetime.h:6:5: error: ?PyTypeObject? does > not name a type; did you mean ?PyType_Check?? > ? ? ?PyTypeObject *TimeType; > ? ? ?^~~~~~~~~~~~ > ? ? ?PyType_Check > /opt/pypy/include/cpyext_datetime.h:7:5: error: ?PyTypeObject? does > not name a type; did you mean ?PyType_Check?? > ? ? ?PyTypeObject *DeltaType; > ? ? ?^~~~~~~~~~~~ > ? ? ?PyType_Check > /opt/pypy/include/cpyext_datetime.h:8:5: error: ?PyTypeObject? does > not name a type; did you mean ?PyType_Check?? > ? ? ?PyTypeObject *TZInfoType; > ? ? ?^~~~~~~~~~~~ > ? ? ?PyType_Check > /opt/pypy/include/cpyext_datetime.h:11:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *(*Date_FromDate)(int, int, int, PyTypeObject*); > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:12:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *(*DateTime_FromDateAndTime)(int, int, int, int, int, > int, int, > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:14:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *(*Time_FromTime)(int, int, int, int, PyObject*, > PyTypeObject*); > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:15:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *(*Delta_FromDelta)(int, int, int, int, PyTypeObject*); > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:18:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *(*DateTime_FromTimestamp)(PyObject*, PyObject*, PyObject*); > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:19:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *(*Date_FromTimestamp)(PyObject*, PyObject*); > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:24:5: error: ?PyObject_HEAD? does > not name a type > ? ? ?PyObject_HEAD > ? ? ?^~~~~~~~~~~~~ > /opt/pypy/include/cpyext_datetime.h:24:5: note: the macro > ?PyObject_HEAD? had not yet been defined > In file included from /opt/pypy/include/object.h:10, > ? ? ? ? ? ? ? ? ?from /opt/pypy/include/Python.h:79, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/platform.h:23, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/pyarrow.h:20, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/common.h:24, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:24 > : > /opt/pypy/include/cpyext_object.h:5: note: it was later defined here > ?#define PyObject_HEAD ?\ > > In file included from /opt/pypy/include/datetime.h:7, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:22 > : > /opt/pypy/include/cpyext_datetime.h:35:5: error: ?PyObject_HEAD? does > not name a type > ? ? ?PyObject_HEAD > ? ? ?^~~~~~~~~~~~~ > /opt/pypy/include/cpyext_datetime.h:35:5: note: the macro > ?PyObject_HEAD? had not yet been defined > In file included from /opt/pypy/include/object.h:10, > ? ? ? ? ? ? ? ? ?from /opt/pypy/include/Python.h:79, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/platform.h:23, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/pyarrow.h:20, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/common.h:24, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:24 > : > /opt/pypy/include/cpyext_object.h:5: note: it was later defined here > ?#define PyObject_HEAD ?\ > > In file included from /opt/pypy/include/datetime.h:7, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:22 > : > /opt/pypy/include/cpyext_datetime.h:37:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *tzinfo; > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:42:5: error: ?PyObject_HEAD? does > not name a type > ? ? ?PyObject_HEAD > ? ? ?^~~~~~~~~~~~~ > /opt/pypy/include/cpyext_datetime.h:42:5: note: the macro > ?PyObject_HEAD? had not yet been defined > In file included from /opt/pypy/include/object.h:10, > ? ? ? ? ? ? ? ? ?from /opt/pypy/include/Python.h:79, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/platform.h:23, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/pyarrow.h:20, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/common.h:24, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:24 > : > /opt/pypy/include/cpyext_object.h:5: note: it was later defined here > ?#define PyObject_HEAD ?\ > > In file included from /opt/pypy/include/datetime.h:7, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:22 > : > /opt/pypy/include/cpyext_datetime.h:44:5: error: ?PyObject? does not > name a type; did you mean ?PyObject_New?? > ? ? ?PyObject *tzinfo; > ? ? ?^~~~~~~~ > ? ? ?PyObject_New > /opt/pypy/include/cpyext_datetime.h:49:5: error: ?PyObject_HEAD? does > not name a type > ? ? ?PyObject_HEAD > ? ? ?^~~~~~~~~~~~~ > /opt/pypy/include/cpyext_datetime.h:49:5: note: the macro > ?PyObject_HEAD? had not yet been defined > In file included from /opt/pypy/include/object.h:10, > ? ? ? ? ? ? ? ? ?from /opt/pypy/include/Python.h:79, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/platform.h:23, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/pyarrow.h:20, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/common.h:24, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:24 > : > /opt/pypy/include/cpyext_object.h:5: note: it was later defined here > ?#define PyObject_HEAD ?\ > > In file included from /opt/pypy/include/datetime.h:7, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:22 > : > /opt/pypy/include/cpyext_datetime.h:54:5: error: ?PyObject_HEAD? does > not name a type > ? ? ?PyObject_HEAD > ? ? ?^~~~~~~~~~~~~ > /opt/pypy/include/cpyext_datetime.h:54:5: note: the macro > ?PyObject_HEAD? had not yet been defined > In file included from /opt/pypy/include/object.h:10, > ? ? ? ? ? ? ? ? ?from /opt/pypy/include/Python.h:79, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/platform.h:23, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/pyarrow.h:20, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/common.h:24, > ? ? ? ? ? ? ? ? ?from /arrow/cpp/src/arrow/python/datetime.cc:24 > : > /opt/pypy/include/cpyext_object.h:5: note: it was later defined here > ?#define PyObject_HEAD ?\ > > In file included from /arrow/cpp/src/arrow/python/datetime.cc:22 > : > /opt/pypy/include/datetime.h:9:30: error: expected constructor, > destructor, or type conversion before ?PyDateTimeAPI? > ?PyAPI_DATA(PyDateTime_CAPI*) PyDateTimeAPI; > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^~~~~~~~~~~~~ > /arrow/cpp/src/arrow/python/datetime.cc:37 :1: > error: expected ?)? before ?=? token > ?PyDateTime_IMPORT > ?^~~~~~~~~~~~~~~~~ > /arrow/cpp/src/arrow/python/datetime.cc : In > function ?arrow::Status arrow::py::internal::PyTime_from_int(int64_t, > arrow::TimeUnit::type, PyObject**)?: > /arrow/cpp/src/arrow/python/datetime.cc:237 > :10: error: ?struct PyDateTime_CAPI? has no > member named ?Time_FromTime? > ? ?*out = PyTime_FromTime(static_cast(hour), > static_cast(minute), > ? ? ? ? ? ^~~~~~~~~~~~~~~ > /arrow/cpp/src/arrow/python/datetime.cc:237 > :10: error: ?struct PyDateTime_CAPI? has no > member named ?TimeType? > ? ?*out = PyTime_FromTime(static_cast(hour), > static_cast(minute), > ? ? ? ? ? ^~~~~~~~~~~~~~~ > /arrow/cpp/src/arrow/python/datetime.cc : In > function ?arrow::Status arrow::py::internal::PyDate_from_int(int64_t, > arrow::DateUnit, PyObject**)?: > /arrow/cpp/src/arrow/python/datetime.cc:245 > :10: error: ?struct PyDateTime_CAPI? has no > member named ?Date_FromDate? > ? ?*out = PyDate_FromDate(static_cast(year), > static_cast(month), > ? ? ? ? ? ^~~~~~~~~~~~~~~ > /arrow/cpp/src/arrow/python/datetime.cc:245 > :10: error: ?struct PyDateTime_CAPI? has no > member named ?DateType? > ? ?*out = PyDate_FromDate(static_cast(year), > static_cast(month), > ? ? ? ? ? ^~~~~~~~~~~~~~~ > /arrow/cpp/src/arrow/python/datetime.cc : In > function ?arrow::Status > arrow::py::internal::PyDateTime_from_int(int64_t, > arrow::TimeUnit::type, PyObject**)?: > /arrow/cpp/src/arrow/python/datetime.cc:257 > :10: error: ?struct PyDateTime_CAPI? has no > member named ?DateTime_FromDateAndTime? > ? ?*out = PyDateTime_FromDateAndTime( > ? ? ? ? ? ^~~~~~~~~~~~~~~~~~~~~~~~~~ > /arrow/cpp/src/arrow/python/datetime.cc:257 > :10: error: ?struct PyDateTime_CAPI? has no > member named ?DateTimeType? > ? ?*out = PyDateTime_FromDateAndTime( > ? ? ? ? ? ^~~~~~~~~~~~~~~~~~~~~~~~~~ > make[2]: *** > [src/arrow/python/CMakeFiles/arrow_python_objlib.dir/build.make:121: > src/arrow/python/CMakeFiles/arrow_python_objlib.dir/datetime.cc.o] Error 1 > make[1]: *** [CMakeFiles/Makefile2:1628: > src/arrow/python/CMakeFiles/arrow_python_objlib.dir/all] Error > > > >> On 10 Sep 2020, at 13:30, Niklas B > > wrote: >> >> That?s what I figured, cool, I will try it! >> >>> On 10 Sep 2020, at 13:28, Matti Picus >> > wrote: >>> >>> >> > From niklas.bivald at enplore.com Mon Sep 14 12:29:10 2020 From: niklas.bivald at enplore.com (Niklas B) Date: Mon, 14 Sep 2020 18:29:10 +0200 Subject: [pypy-dev] PyArrow on PyPy: PyDateTime_CAPI question (getting cpyext to work) in c++ In-Reply-To: References: <1ECAEE19-B13B-4403-9E8F-1ACDCFF2E113@enplore.com> <88054D70-885B-4E03-8F09-FE4435CFCAC8@enplore.com> Message-ID: <24CCED01-6542-418D-A392-5F44250E07E6@enplore.com> Yep, I can confirm your patch builds and works for parquet the very least (https://github.com/bivald/pyarrow-on-pypy3/tree/feature/latest-pypy-latest-pyarrow ) I would say it works surprisingly well for parquet: ======================================================================================= test session starts ======================================================================================== platform linux -- Python 3.6.9[pypy-7.3.3-alpha], pytest-6.0.2, py-1.9.0, pluggy-0.13.1 rootdir: /arrow/python, configfile: setup.cfg plugins: hypothesis-5.35.1 collected 286 items pyarrow2/tests/test_parquet.py .s.s.s.s..s.s.s.s.ss.s.s.s.s.s.s..s.s.s.s.s.s.s.s.s.s..s.s..s.s.................................s.s...........s..s.s.sxs.s.s.sxs.sxsss....s.s.s.s....s.s.s.s. [ 54%] s.sxsxs.s...s.s.s.s.s.s..s.ssssssss.s.s.s.s.s.s.s.sx.s.s.s....s..s.s..s.s.s..s.s.s.s..s.s...ss..s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.sss [100%] =========================================================================== 164 passed, 116 skipped, 6 xfailed in 43.69s =========================================================================== Overall the entire test suite: ====================================================================================================== short test summary info ====================================================================================================== FAILED pyarrow2/tests/test_array.py::test_to_pandas_zero_copy - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_array_slice - SystemError: Function returned an error result without setting an exception FAILED pyarrow2/tests/test_array.py::test_array_ref_to_ndarray_base - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_array_conversions_no_sentinel_values - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_array.py::test_nbytes_sizeof - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_cffi.py::test_export_import_array - assert 1528 == 896 FAILED pyarrow2/tests/test_cffi.py::test_export_import_batch - assert 1048 == 128 FAILED pyarrow2/tests/test_convert_builtin.py::test_garbage_collection - assert 128 == 766912 FAILED pyarrow2/tests/test_convert_builtin.py::test_sequence_bytes - NotImplementedError: creating contiguous readonly buffer from non-contiguous not implemented yet FAILED pyarrow2/tests/test_convert_builtin.py::test_map_from_dicts - AssertionError: Regex pattern 'integer is required' does not match 'expected integer, got str object'. FAILED pyarrow2/tests/test_csv.py::test_read_options - Failed: DID NOT RAISE FAILED pyarrow2/tests/test_csv.py::test_parse_options - Failed: DID NOT RAISE FAILED pyarrow2/tests/test_csv.py::test_convert_options - Failed: DID NOT RAISE FAILED pyarrow2/tests/test_csv.py::TestSerialStreamingCSVRead::test_batch_lifetime - AssertionError: assert 1464704 == 1464576 FAILED pyarrow2/tests/test_cython.py::test_cython_api - subprocess.CalledProcessError: Command '['/pyarrow/bin/pypy3', 'setup.py', 'build_ext', '--inplace']' returned non-zero exit status 1. FAILED pyarrow2/tests/test_extension_type.py::test_ext_type__lifetime - AssertionError: assert UuidType(extension) is None FAILED pyarrow2/tests/test_extension_type.py::test_uuid_type_pickle - AssertionError: assert UuidType(extension) is None FAILED pyarrow2/tests/test_extension_type.py::test_ext_array_lifetime - AssertionError: assert ParamExtType(extension) is None FAILED pyarrow2/tests/test_fs.py::test_py_filesystem_lifetime - AssertionError: assert is None FAILED pyarrow2/tests/test_pandas.py::test_to_pandas_deduplicate_integers_as_objects - assert 100 == 991 FAILED pyarrow2/tests/test_pandas.py::test_array_uses_memory_pool - assert 103552 == 465152 FAILED pyarrow2/tests/test_pandas.py::test_to_pandas_self_destruct - assert 6112064 == 4112064 FAILED pyarrow2/tests/test_pandas.py::test_table_uses_memory_pool - assert 6249408 == 6112064 FAILED pyarrow2/tests/test_pandas.py::test_object_leak_in_numpy_array - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_pandas.py::test_object_leak_in_dataframe - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_schema.py::test_schema_sizeof - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_coo_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_csr_matrix_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_sparse_tensor.py::test_sparse_csf_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' FAILED pyarrow2/tests/test_table.py::test_chunked_array_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_table.py::test_recordbatch_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_table.py::test_table_basics - TypeError: getsizeof(...) FAILED pyarrow2/tests/test_tensor.py::test_tensor_base_object - AttributeError: module 'sys' has no attribute 'getrefcount' ========================================================================= 33 failed, 2620 passed, 532 skipped, 13 xfailed, 10 warnings in 104.02s (0:01:44) ========================================================================= Segfaults on a couple of the tests, but a lot passes as well > On 11 Sep 2020, at 14:18, Matti Picus wrote: > > Thse changes seems to compile for me, but I did not run the tests. > > https://gist.github.com/mattip/c9c8398b58721ae5893dc8134c353f28 > > Matti > > > On 9/11/20 1:01 PM, Niklas B wrote: >> The PyMemoryView_GetContiguous errors are all gone - good work! >> >> It didn?t really like my butchering of datetime.cc https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python/datetime.cc#L37 >> >> Added: >> #include >> >> And >> PyDateTime_IMPORT >> >> >> >> Then changed: >> datetime_api = >> >> reinterpret_cast(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)); >> >> >> To: >> >> datetime_api = PyDateTimeAPI; >> >> They do some PyDateTimeAPI voodoo at https://github.com/apache/arrow/blob/apache-arrow-1.0.1/cpp/src/arrow/python/datetime.h#L29 which might be a culprit as well. >> >> >> >> >> >> >> >> >> >> Gives the following outputs: >> /arrow/cpp/src/arrow/python/datetime.h:34: warning: "PyDateTimeAPI" redefined >> #define PyDateTimeAPI ::arrow::py::internal::datetime_api >> >> In file included from /opt/pypy/include/Python.h:144, >> from /arrow/cpp/src/arrow/python/platform.h:23, >> from /arrow/cpp/src/arrow/python/pyarrow.h:20, >> from /arrow/cpp/src/arrow/python/common.h:24, >> from /arrow/cpp/src/arrow/python/datetime.cc:24 : >> /opt/pypy/include/pypy_decl.h:1121: note: this is the location of the previous definition >> #define PyDateTimeAPI PyPyDateTimeAPI >> >> In file included from /opt/pypy/include/datetime.h:7, >> from /arrow/cpp/src/arrow/python/datetime.cc:22 : >> /opt/pypy/include/cpyext_datetime.h:4:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? >> PyTypeObject *DateType; >> ^~~~~~~~~~~~ >> PyType_Check >> /opt/pypy/include/cpyext_datetime.h:5:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? >> PyTypeObject *DateTimeType; >> ^~~~~~~~~~~~ >> PyType_Check >> /opt/pypy/include/cpyext_datetime.h:6:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? >> PyTypeObject *TimeType; >> ^~~~~~~~~~~~ >> PyType_Check >> /opt/pypy/include/cpyext_datetime.h:7:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? >> PyTypeObject *DeltaType; >> ^~~~~~~~~~~~ >> PyType_Check >> /opt/pypy/include/cpyext_datetime.h:8:5: error: ?PyTypeObject? does not name a type; did you mean ?PyType_Check?? >> PyTypeObject *TZInfoType; >> ^~~~~~~~~~~~ >> PyType_Check >> /opt/pypy/include/cpyext_datetime.h:11:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *(*Date_FromDate)(int, int, int, PyTypeObject*); >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:12:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *(*DateTime_FromDateAndTime)(int, int, int, int, int, int, int, >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:14:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *(*Time_FromTime)(int, int, int, int, PyObject*, PyTypeObject*); >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:15:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *(*Delta_FromDelta)(int, int, int, int, PyTypeObject*); >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:18:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *(*DateTime_FromTimestamp)(PyObject*, PyObject*, PyObject*); >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:19:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *(*Date_FromTimestamp)(PyObject*, PyObject*); >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:24:5: error: ?PyObject_HEAD? does not name a type >> PyObject_HEAD >> ^~~~~~~~~~~~~ >> /opt/pypy/include/cpyext_datetime.h:24:5: note: the macro ?PyObject_HEAD? had not yet been defined >> In file included from /opt/pypy/include/object.h:10, >> from /opt/pypy/include/Python.h:79, >> from /arrow/cpp/src/arrow/python/platform.h:23, >> from /arrow/cpp/src/arrow/python/pyarrow.h:20, >> from /arrow/cpp/src/arrow/python/common.h:24, >> from /arrow/cpp/src/arrow/python/datetime.cc:24 : >> /opt/pypy/include/cpyext_object.h:5: note: it was later defined here >> #define PyObject_HEAD \ >> >> In file included from /opt/pypy/include/datetime.h:7, >> from /arrow/cpp/src/arrow/python/datetime.cc:22 : >> /opt/pypy/include/cpyext_datetime.h:35:5: error: ?PyObject_HEAD? does not name a type >> PyObject_HEAD >> ^~~~~~~~~~~~~ >> /opt/pypy/include/cpyext_datetime.h:35:5: note: the macro ?PyObject_HEAD? had not yet been defined >> In file included from /opt/pypy/include/object.h:10, >> from /opt/pypy/include/Python.h:79, >> from /arrow/cpp/src/arrow/python/platform.h:23, >> from /arrow/cpp/src/arrow/python/pyarrow.h:20, >> from /arrow/cpp/src/arrow/python/common.h:24, >> from /arrow/cpp/src/arrow/python/datetime.cc:24 : >> /opt/pypy/include/cpyext_object.h:5: note: it was later defined here >> #define PyObject_HEAD \ >> >> In file included from /opt/pypy/include/datetime.h:7, >> from /arrow/cpp/src/arrow/python/datetime.cc:22 : >> /opt/pypy/include/cpyext_datetime.h:37:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *tzinfo; >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:42:5: error: ?PyObject_HEAD? does not name a type >> PyObject_HEAD >> ^~~~~~~~~~~~~ >> /opt/pypy/include/cpyext_datetime.h:42:5: note: the macro ?PyObject_HEAD? had not yet been defined >> In file included from /opt/pypy/include/object.h:10, >> from /opt/pypy/include/Python.h:79, >> from /arrow/cpp/src/arrow/python/platform.h:23, >> from /arrow/cpp/src/arrow/python/pyarrow.h:20, >> from /arrow/cpp/src/arrow/python/common.h:24, >> from /arrow/cpp/src/arrow/python/datetime.cc:24 : >> /opt/pypy/include/cpyext_object.h:5: note: it was later defined here >> #define PyObject_HEAD \ >> >> In file included from /opt/pypy/include/datetime.h:7, >> from /arrow/cpp/src/arrow/python/datetime.cc:22 : >> /opt/pypy/include/cpyext_datetime.h:44:5: error: ?PyObject? does not name a type; did you mean ?PyObject_New?? >> PyObject *tzinfo; >> ^~~~~~~~ >> PyObject_New >> /opt/pypy/include/cpyext_datetime.h:49:5: error: ?PyObject_HEAD? does not name a type >> PyObject_HEAD >> ^~~~~~~~~~~~~ >> /opt/pypy/include/cpyext_datetime.h:49:5: note: the macro ?PyObject_HEAD? had not yet been defined >> In file included from /opt/pypy/include/object.h:10, >> from /opt/pypy/include/Python.h:79, >> from /arrow/cpp/src/arrow/python/platform.h:23, >> from /arrow/cpp/src/arrow/python/pyarrow.h:20, >> from /arrow/cpp/src/arrow/python/common.h:24, >> from /arrow/cpp/src/arrow/python/datetime.cc:24 : >> /opt/pypy/include/cpyext_object.h:5: note: it was later defined here >> #define PyObject_HEAD \ >> >> In file included from /opt/pypy/include/datetime.h:7, >> from /arrow/cpp/src/arrow/python/datetime.cc:22 : >> /opt/pypy/include/cpyext_datetime.h:54:5: error: ?PyObject_HEAD? does not name a type >> PyObject_HEAD >> ^~~~~~~~~~~~~ >> /opt/pypy/include/cpyext_datetime.h:54:5: note: the macro ?PyObject_HEAD? had not yet been defined >> In file included from /opt/pypy/include/object.h:10, >> from /opt/pypy/include/Python.h:79, >> from /arrow/cpp/src/arrow/python/platform.h:23, >> from /arrow/cpp/src/arrow/python/pyarrow.h:20, >> from /arrow/cpp/src/arrow/python/common.h:24, >> from /arrow/cpp/src/arrow/python/datetime.cc:24 : >> /opt/pypy/include/cpyext_object.h:5: note: it was later defined here >> #define PyObject_HEAD \ >> >> In file included from /arrow/cpp/src/arrow/python/datetime.cc:22 : >> /opt/pypy/include/datetime.h:9:30: error: expected constructor, destructor, or type conversion before ?PyDateTimeAPI? >> PyAPI_DATA(PyDateTime_CAPI*) PyDateTimeAPI; >> ^~~~~~~~~~~~~ >> /arrow/cpp/src/arrow/python/datetime.cc:37 :1: error: expected ?)? before ?=? token >> PyDateTime_IMPORT >> ^~~~~~~~~~~~~~~~~ >> /arrow/cpp/src/arrow/python/datetime.cc : In function ?arrow::Status arrow::py::internal::PyTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)?: >> /arrow/cpp/src/arrow/python/datetime.cc:237 :10: error: ?struct PyDateTime_CAPI? has no member named ?Time_FromTime? >> *out = PyTime_FromTime(static_cast(hour), static_cast(minute), >> ^~~~~~~~~~~~~~~ >> /arrow/cpp/src/arrow/python/datetime.cc:237 :10: error: ?struct PyDateTime_CAPI? has no member named ?TimeType? >> *out = PyTime_FromTime(static_cast(hour), static_cast(minute), >> ^~~~~~~~~~~~~~~ >> /arrow/cpp/src/arrow/python/datetime.cc : In function ?arrow::Status arrow::py::internal::PyDate_from_int(int64_t, arrow::DateUnit, PyObject**)?: >> /arrow/cpp/src/arrow/python/datetime.cc:245 :10: error: ?struct PyDateTime_CAPI? has no member named ?Date_FromDate? >> *out = PyDate_FromDate(static_cast(year), static_cast(month), >> ^~~~~~~~~~~~~~~ >> /arrow/cpp/src/arrow/python/datetime.cc:245 :10: error: ?struct PyDateTime_CAPI? has no member named ?DateType? >> *out = PyDate_FromDate(static_cast(year), static_cast(month), >> ^~~~~~~~~~~~~~~ >> /arrow/cpp/src/arrow/python/datetime.cc : In function ?arrow::Status arrow::py::internal::PyDateTime_from_int(int64_t, arrow::TimeUnit::type, PyObject**)?: >> /arrow/cpp/src/arrow/python/datetime.cc:257 :10: error: ?struct PyDateTime_CAPI? has no member named ?DateTime_FromDateAndTime? >> *out = PyDateTime_FromDateAndTime( >> ^~~~~~~~~~~~~~~~~~~~~~~~~~ >> /arrow/cpp/src/arrow/python/datetime.cc:257 :10: error: ?struct PyDateTime_CAPI? has no member named ?DateTimeType? >> *out = PyDateTime_FromDateAndTime( >> ^~~~~~~~~~~~~~~~~~~~~~~~~~ >> make[2]: *** [src/arrow/python/CMakeFiles/arrow_python_objlib.dir/build.make:121: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/datetime.cc.o] Error 1 >> make[1]: *** [CMakeFiles/Makefile2:1628: src/arrow/python/CMakeFiles/arrow_python_objlib.dir/all] Error >> >> >> >>> On 10 Sep 2020, at 13:30, Niklas B > wrote: >>> >>> That?s what I figured, cool, I will try it! >>> >>>> On 10 Sep 2020, at 13:28, Matti Picus > wrote: >>>> >>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Sep 16 09:56:30 2020 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 16 Sep 2020 16:56:30 +0300 Subject: [pypy-dev] time for a new release: rc2 Message-ID: <8946bcd2-429d-90e5-a770-d76d3f6326ea@gmail.com> I have uploaded rc2 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ (note the trailing slash) The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download_advanced.rst#L465 The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html This release does include a 3.7 alpha. Please try them out, especially on windows (extra points for non-english interfaces and install paths) and macos (extra points for machines that run without homebrew stuff installed), to make sure you can run your project with them. Any comments are welcome. Since the rc1 candidates, we have updated the stdlibs of 3.6 and 2.7 for various upstream issues and CVEs, and fixed an issue with rpython on ARM. Thanks to all who helped test and fix. 3.7 now uses stdlib 3.7.9. Matti From mgorny at gentoo.org Thu Sep 17 18:48:38 2020 From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) Date: Fri, 18 Sep 2020 00:48:38 +0200 Subject: [pypy-dev] time for a new release: rc2 In-Reply-To: <8946bcd2-429d-90e5-a770-d76d3f6326ea@gmail.com> References: <8946bcd2-429d-90e5-a770-d76d3f6326ea@gmail.com> Message-ID: On Wed, 2020-09-16 at 16:56 +0300, Matti Picus wrote: > I have uploaded rc2 of pypy v7.3.2 to https://buildbot.pypy.org/pypy/ > (note the trailing slash) > > > The hashes are here > https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download_advanced.rst#L465 > > > The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html > > > This release does include a 3.7 alpha. > Ok, I've done some thorough testing today. PyPy2.7 looks all good. PyPy3.6 has some test failures, including 2 that are not on buildbot but that's not a regression from previous version. PyPy3.7 has lots of test failures, too many for me to compare against buildbot. I've also tested PyPy3.7 against a set of ~200 packages. Two specific problems came up: 1. attrs and zope.interface both fail tests with the following exception: zope.interface.exceptions.InvalidInterface: Concrete attribute, __orig_bases__ 2. hypothesis fails tests with asyncio error: Traceback (most recent call last): File "/tmp/portage/dev-python/hypothesis-5.35.3/work/hypothesis-hypothesis-python-5.35.3/hypothesis-python-pypy3/tests/cover/test_async_def.py", line 33, in test_foo @given(st.text()) File "/tmp/portage/dev-python/hypothesis-5.35.3/work/hypothesis-hypothesis-python-5.35.3/hypothesis-python-pypy3/build/lib/hypothesis/core.py", line 1145, in wrapped_test raise the_error_hypothesis_found File "/tmp/portage/dev-python/hypothesis-5.35.3/work/hypothesis-hypothesis-python-5.35.3/hypothesis-python-pypy3/tests/cover/test_async_def.py", line 30, in execute_example asyncio.run(f()) File "/usr/lib/pypy3.7/lib-python/3/asyncio/runners.py", line 39, in run loop = events.new_event_loop() File "/usr/lib/pypy3.7/lib-python/3/asyncio/events.py", line 762, in new_event_loop return get_event_loop_policy().new_event_loop() File "/usr/lib/pypy3.7/lib-python/3/asyncio/events.py", line 660, in new_event_loop return self._loop_factory() File "/usr/lib/pypy3.7/lib-python/3/asyncio/unix_events.py", line 51, in __init__ super().__init__(selector) File "/usr/lib/pypy3.7/lib-python/3/asyncio/selector_events.py", line 54, in __init__ super().__init__() File "/usr/lib/pypy3.7/lib-python/3/asyncio/base_events.py", line 370, in __init__ self._clock_resolution = time.get_clock_info('monotonic').resolution TypeError: time() takes 0 positional arguments but 1 was given -- Best regards, Micha? G?rny -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: This is a digitally signed message part URL: From anto.cuni at gmail.com Sat Sep 19 17:50:09 2020 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 19 Sep 2020 23:50:09 +0200 Subject: [pypy-dev] Interaction between HPy_CAST and HPy_AsPyObject Message-ID: Consider the following snippet of code: typedef struct { HPyObject_HEAD long x; long y; } PointObject; void foo(HPyContext ctx, HPy h_point) { PointObject *p1 = HPy_CAST(ctx, PointObject, h_point); PyObjecy *py_point = HPy_AsPyObject(ctx, h_point); // [1] PointObject *p2 = (PointObject*)py_point; ... } [1] Note that it does not need to be a call to HPy_AsPyObject: it might be a legacy method which takes a PyObject *self, or other similar ways It is obvious that HPy_CAST and HPy_AsPyObject need to return the very same address. This is straightforward to implement on CPython, but it poses some challenges on PyPy (and probably GraalPython). Things to consider: 1. currently, in PyPy we allocate the PointObject at a non-movable address, but so far the API does not REQUIRE it. I think it would be reasonable to have an implementation in which objects are movable and HPy_CAST pins the memory until the originating handle is closed. OTOH, the only reasonable semantics is that multiple calls to HPy_AsPyObject returns always the same address. 2. HPyObject_HEAD consists of two words which can be used by the implementation as they like. On CPython, it is obviously mapped to PyObject_HEAD, but in PyPy we (usually) don't need these two extra words, so we allocate sizeof(PointObject)-16 and return a pointer to malloc()-16, which works well since nobody is accessing those two words. I think that GraalPython could use a similar approach. 3. On PyPy, PyObject_HEAD is *three words*, because it also contains ob_pypy_link. But, since the code uses *H*PyObject_HEAD, PointObject will contain only 2 extra words. 4. In the real world usage, there will be "pure hpy types" and "legacy hpy types", which uses legacy methods&co. It would be nice if the pure hpy types do NOT have to pay penalties in case they are never casted to PyObject* With this in mind, how do we implement HPy_AsPyObject on PyPy? One easy way is: 1. we allocate sizeof(PointObject)+8 2. we tweak cpyext to find ob_pypy_link at p-8 3. we teach cpyext how to convert W_HPyObject into PyObject* and vice versa. However, this means that we need to always allocate 24 extra bytes for each object, even if nobody ever calls HPy_AsPyObject on it, which looks bad. Moreover, without changes in the API, the pin/unpin implementation of HPy_CAST becomes de facto impossible. So, my proposal is to distinguish between "legacy hpy types" and "pure hpy types". An HPyType_Spec is legacy if: 1. it uses .legacy_slots = ... OR 2. it ses .legacy = true (i.e., you can explicitly mark a type as legacy even if you no longer have any legacy method/slot. This is useful if you pass it to ANOTHER type which expects to be able to cast the PyObject* into the struct). If a type is "legacy", the snippet shown above works as expected; if it's not legacy, it is still possible to call HPy_AsPyObject on it, but then you are no longer allowed to C-cast if to PointObject* (on pypy, this will mean that you will get a "standard" PyObject* which is a proxy to W_HPyObject). Ideally, in that case it would be nice to catch the invalid cast in the debug mode, but I don't think this is possible... too bad. What do you think? ciao, Anto -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Sun Sep 20 17:56:57 2020 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 20 Sep 2020 23:56:57 +0200 Subject: [pypy-dev] Interaction between HPy_CAST and HPy_AsPyObject In-Reply-To: References: Message-ID: After an IRC discussion with Armin, we designed the following solution: https://github.com/hpyproject/hpy/issues/83 If you have comments, please post the on the github issue, to avoid splitting discussions half here and half there :) ciao, Antonio On Sat, Sep 19, 2020 at 11:50 PM Antonio Cuni wrote: > Consider the following snippet of code: > > typedef struct { > HPyObject_HEAD > long x; > long y; > } PointObject; > void foo(HPyContext ctx, HPy h_point) > { > PointObject *p1 = HPy_CAST(ctx, PointObject, h_point); > PyObjecy *py_point = HPy_AsPyObject(ctx, h_point); // [1] > PointObject *p2 = (PointObject*)py_point; > ... > } > > [1] Note that it does not need to be a call to HPy_AsPyObject: it might > be a legacy method which takes a PyObject *self, or other similar ways > > > It is obvious that HPy_CAST and HPy_AsPyObject need to return the very > same address. This is straightforward to implement on CPython, but it poses > some challenges on PyPy (and probably GraalPython). > > > Things to consider: > > 1. currently, in PyPy we allocate the PointObject at a non-movable > address, but so far the API does not REQUIRE it. I think it would be > reasonable to have an implementation in which objects are movable and > HPy_CAST pins the memory until the originating handle is closed. OTOH, the > only reasonable semantics is that multiple calls to HPy_AsPyObject returns > always the same address. > > > 2. HPyObject_HEAD consists of two words which can be used by the > implementation as they like. On CPython, it is obviously mapped to > PyObject_HEAD, but in PyPy we (usually) don't need these two extra words, > so we allocate sizeof(PointObject)-16 and return a pointer to malloc()-16, > which works well since nobody is accessing those two words. I think that > GraalPython could use a similar approach. > > > 3. On PyPy, PyObject_HEAD is *three words*, because it also contains > ob_pypy_link. But, since the code uses *H*PyObject_HEAD, PointObject will > contain only 2 extra words. > > > 4. In the real world usage, there will be "pure hpy types" and "legacy hpy > types", which uses legacy methods&co. It would be nice if the pure hpy > types do NOT have to pay penalties in case they are never casted to > PyObject* > > > > With this in mind, how do we implement HPy_AsPyObject on PyPy? One easy > way is: > > 1. we allocate sizeof(PointObject)+8 > > 2. we tweak cpyext to find ob_pypy_link at p-8 > > 3. we teach cpyext how to convert W_HPyObject into PyObject* and vice > versa. > > > However, this means that we need to always allocate 24 extra bytes for > each object, even if nobody ever calls HPy_AsPyObject on it, which looks > bad. Moreover, without changes in the API, the pin/unpin implementation of > HPy_CAST becomes de facto impossible. > > > So, my proposal is to distinguish between "legacy hpy types" and "pure hpy > types". An HPyType_Spec is legacy if: > > 1. it uses .legacy_slots = ... OR > > 2. it ses .legacy = true (i.e., you can explicitly mark a type as legacy > even if you no longer have any legacy method/slot. This is useful if you > pass it to ANOTHER type which expects to be able to cast the PyObject* into > the struct). > > > If a type is "legacy", the snippet shown above works as expected; if it's > not legacy, it is still possible to call HPy_AsPyObject on it, but then you > are no longer allowed to C-cast if to PointObject* (on pypy, this will mean > that you will get a "standard" PyObject* which is a proxy to W_HPyObject). > > Ideally, in that case it would be nice to catch the invalid cast in the > debug mode, but I don't think this is possible... too bad. > > > What do you think? > > ciao, > > Anto > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Mon Sep 21 05:19:33 2020 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 21 Sep 2020 11:19:33 +0200 Subject: [pypy-dev] PyPy hpy branch workflow Message-ID: Hi pypy-dev (cc:ing hpy-dev), I have just merged the hpy branch into py3.6. This means that from now, pypy3.6 nightly builds will automatically support hpy, and it will be much easier for interested users to try it out. I propose the following workflow for continuing HPy development on PyPy: 1. the development will continue on the hpy branch, in "update_vendored" steps: i.e., periodically we run the script to update to a newer hpy version and implement all the new features. As soon as we do it, new tests are introduced and they (hopefully :)) will start failing 2. note that it is not necessarily to ./update_vendored.sh to the LATEST git version: this will probably introduce too many features and will make it harder/longer to make all tests green again. So I suggest to gradually ./update_vendored.sh in smaller step (you can do it by doing "git checkout REV" in your main hpy working copy before running update_vendored) 3. Once the tests are green again, we can merge hpy into py3.6. The merge can happen directly or with a gitlab MR if you want someone to review the code. Personally, I volunteer to review all hpy-related MR, so feel free to ping me if you want :) 4. goto 1 At the moment, all tests inside module/_hpy_universal pass, so we are in a green state: I would like to try hard to keep them green, and merge changes only after the tests pass: the hpy tests are run automatically on gitlab-ci whenever you push to the hpy branch, so it should be doable. A note about nightly builds: as soon as we catch up with hpy git revision 0a46d31, it will be possible to run "import hpy; hpy.get_version()". This will be very useful for nightly builds, because it will tell you exactly which hpy.devel revision to check out in order to be compatible with your nightly build. As soon as the API stabilizes we will want to use official version numbers, but I think that for now it's a reasonable approach. Please let me know if you have any questions. ciao, Anto -------------- next part -------------- An HTML attachment was scrubbed... URL: From phyo.arkarlwin at gmail.com Wed Sep 23 11:22:09 2020 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Wed, 23 Sep 2020 21:52:09 +0630 Subject: [pypy-dev] Changing the PyPy download page In-Reply-To: References: Message-ID: the image is out of place , on 4k laptop. [image: image.png] On Sun, Aug 30, 2020 at 7:31 PM Ram Rachum wrote: > Hi everyone, > > Just a follow-up on this thread: Matti and I overhauled the PyPy download > experience. > > This includes the download button on the main page: https://www.pypy.org/ > and the new short-and-to-the-point download page. > > > Thanks, > Ram. > > On Mon, Jun 29, 2020, 16:01 Ram Rachum wrote: > >> Awesome. Thanks for the support guys. I've put this on my todo list. >> >> On Mon, Jun 29, 2020 at 4:00 PM David Edelsohn wrote: >> >>> On Sun, Jun 28, 2020 at 3:36 PM Ram Rachum wrote: >>> > >>> > Hi everyone! >>> > >>> > The last week, I've been talking with Matti about ways in which PyPy >>> could be friendlier to new users, and what to do about that. One of the >>> examples I raised in which PyPy is, in my opinion, giving newbies a hard >>> time, is the download page. >>> > >>> > In my opinion it's way too complicated and not geared for people who >>> want to use PyPy but are less knowledgeable, or less interested in putting >>> in time to understand the subtleties of JIT vs no-JIT vs STM, etc. >>> > >>> > We discussed that maybe I should make that change and open a PR for >>> it. I said I'm willing to do that, (and learn some Mercurial and Nikita on >>> the way) if I know there's general support in this list to that direction >>> of change; I expect a code review, but I want to know before I start that >>> this change is wanted. >>> > >>> > Here are a few of the changes I'd like to make: >>> > >>> > Push the list of binaries to the top. >>> > Put Python 3 above Python 2. >>> > Move the instructions for building to a separate page. The >>> intersection of the set "people who are interested in build instructions" >>> with the set "people who have a hard time pressing an additional link to >>> get to the build instruction" is very small indeed. >>> > I might also put icons of Windows, Mac and Linux near their respective >>> binaries. >>> > Ideally I would have auto-detection that gives you the binary to your >>> OS, but I'm not sure I want to work that hard. >>> > >>> > You get the general idea: Treating PyPy more like a finished product >>> and less like a C library. >>> > >>> > What do you think? >>> >>> +1 >>> >>> - David >>> >> _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 357883 bytes Desc: not available URL: From ram at rachum.com Wed Sep 23 11:23:59 2020 From: ram at rachum.com (Ram Rachum) Date: Wed, 23 Sep 2020 18:23:59 +0300 Subject: [pypy-dev] Changing the PyPy download page In-Reply-To: References: Message-ID: That is true and I've seen it on my computer too. However, it's not related to our change. Open a ticket. On Wed, Sep 23, 2020 at 6:22 PM Phyo Arkar wrote: > the image is out of place , on 4k laptop. > > [image: image.png] > > On Sun, Aug 30, 2020 at 7:31 PM Ram Rachum wrote: > >> Hi everyone, >> >> Just a follow-up on this thread: Matti and I overhauled the PyPy download >> experience. >> >> This includes the download button on the main page: https://www.pypy.org/ >> and the new short-and-to-the-point download page. >> >> >> Thanks, >> Ram. >> >> On Mon, Jun 29, 2020, 16:01 Ram Rachum wrote: >> >>> Awesome. Thanks for the support guys. I've put this on my todo list. >>> >>> On Mon, Jun 29, 2020 at 4:00 PM David Edelsohn >>> wrote: >>> >>>> On Sun, Jun 28, 2020 at 3:36 PM Ram Rachum wrote: >>>> > >>>> > Hi everyone! >>>> > >>>> > The last week, I've been talking with Matti about ways in which PyPy >>>> could be friendlier to new users, and what to do about that. One of the >>>> examples I raised in which PyPy is, in my opinion, giving newbies a hard >>>> time, is the download page. >>>> > >>>> > In my opinion it's way too complicated and not geared for people who >>>> want to use PyPy but are less knowledgeable, or less interested in putting >>>> in time to understand the subtleties of JIT vs no-JIT vs STM, etc. >>>> > >>>> > We discussed that maybe I should make that change and open a PR for >>>> it. I said I'm willing to do that, (and learn some Mercurial and Nikita on >>>> the way) if I know there's general support in this list to that direction >>>> of change; I expect a code review, but I want to know before I start that >>>> this change is wanted. >>>> > >>>> > Here are a few of the changes I'd like to make: >>>> > >>>> > Push the list of binaries to the top. >>>> > Put Python 3 above Python 2. >>>> > Move the instructions for building to a separate page. The >>>> intersection of the set "people who are interested in build instructions" >>>> with the set "people who have a hard time pressing an additional link to >>>> get to the build instruction" is very small indeed. >>>> > I might also put icons of Windows, Mac and Linux near their >>>> respective binaries. >>>> > Ideally I would have auto-detection that gives you the binary to your >>>> OS, but I'm not sure I want to work that hard. >>>> > >>>> > You get the general idea: Treating PyPy more like a finished product >>>> and less like a C library. >>>> > >>>> > What do you think? >>>> >>>> +1 >>>> >>>> - David >>>> >>> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 357883 bytes Desc: not available URL: From matti.picus at gmail.com Thu Sep 24 04:13:16 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 24 Sep 2020 11:13:16 +0300 Subject: [pypy-dev] Changing the PyPy download page In-Reply-To: References: Message-ID: <4e9cab08-93f2-a25e-95c2-31481eec5d01@gmail.com> On Wed, Sep 23, 2020 at 6:22 PM Phyo Arkar > wrote: the image is out of place? , on 4k laptop. ... Thanks for the heads up. How about if we rearrange the pieces on the page so that the image is %100 wide and not in a column since it is too big? I will try it the next time I rebuild the site, which will be for the upcoming release. I am not really a UX designer, so please keep the comments and suggestions coming. Matti From matti.picus at gmail.com Thu Sep 24 17:57:28 2020 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 25 Sep 2020 00:57:28 +0300 Subject: [pypy-dev] release is ready Message-ID: I have uploaded pypy v7.3.2, it should be available at https://downloads.python.org/pypy/ The hashes are here https://foss.heptapod.net/pypy/pypy.org/-/blob/branch/default/pages/download_advanced.rst#L465, the updated web site should deploy in an hour or so at https://www.pypy.org/download.html The release note is here https://doc.pypy.org/en/latest/release-v7.3.2.html This release does include a 3.7 alpha. I will be publishing an official release notice in a few hours. Thanks to all who tried out the release candidates. Matti From Trevor.Clack at gdit.com Tue Sep 29 19:15:44 2020 From: Trevor.Clack at gdit.com (Clack, Trevor) Date: Tue, 29 Sep 2020 23:15:44 +0000 Subject: [pypy-dev] Short discussion about pypy overview In-Reply-To: References: Message-ID: Hello, I'm working on giving a presentation to a high performance computing community on pypy. I've got a good deal of content developed except for a high level overview of how pypy in fact works. Several days of YouTube, searching stack overflow, and searching the docs haven't clarified much. I was hoping to get in contact with a developer if it's not too much trouble Trevor Clack HPCMP PET Computational Scientist Mobile 949.412.9902 Trevor.Clack at gdit.com 3909 Halls Ferry Road Vicksburg, MS 39180 www.gdit.com [cid:image001.png at 01D664F5.C0E0A040] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-cid_image0.png Type: image/png Size: 7680 bytes Desc: Outlook-cid_image0.png URL: From matti.picus at gmail.com Wed Sep 30 03:42:18 2020 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 30 Sep 2020 10:42:18 +0300 Subject: [pypy-dev] Short discussion about pypy overview In-Reply-To: References: Message-ID: On 9/30/20 2:15 AM, Clack, Trevor wrote: > Hello, I'm working on giving a presentation to a high performance > computing community on pypy. I've got a good deal of content developed > except for a high level overview of how pypy in fact works. Several > days of YouTube, searching stack overflow, and searching the docs > haven't clarified much. I was hoping to get in contact with a > developer if it's not too much trouble > > Trevor Clack > > HPCMP PET Computational Scientist > > Mobile? 949.412.9902 > > Trevor.Clack at gdit.com > > 3909 Halls Ferry Road > > Vicksburg, MS 39180 > > www.gdit.com > > cid:image001.png at 01D664F5.C0E0A040 > > Hi and welcome. Could you be a little more specific: do you mean "how the RPython tool chain can compile a Python interpreter", "how is PyPy fast?", "what optimizations can the JIT do", "how does a tracing JIT work, "why doesn't PyPy use refcount semantics", or something else? Matti From Trevor.Clack at gdit.com Tue Sep 29 19:13:55 2020 From: Trevor.Clack at gdit.com (Clack, Trevor) Date: Tue, 29 Sep 2020 23:13:55 +0000 Subject: [pypy-dev] Fw: short discussion about pypy overview In-Reply-To: References: Message-ID: Hello, I'm working on giving a presentation to a high performance computing community on pypy. I've got a good deal of content developed except for a high level overview of how pypy in fact works. Several days of YouTube, searching stack overflow, and searching the docs haven't clarified much. I was hoping to get in contact with a developer if it's not too much trouble Trevor Clack HPCMP PET Computational Scientist Mobile 949.412.9902 Trevor.Clack at gdit.com 3909 Halls Ferry Road Vicksburg, MS 39180 www.gdit.com [cid:image001.png at 01D664F5.C0E0A040] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-cid_image0.png Type: image/png Size: 7680 bytes Desc: Outlook-cid_image0.png URL: From Trevor.Clack at gdit.com Wed Sep 30 11:52:58 2020 From: Trevor.Clack at gdit.com (Clack, Trevor) Date: Wed, 30 Sep 2020 15:52:58 +0000 Subject: [pypy-dev] Short discussion about pypy overview In-Reply-To: References: , Message-ID: Hi Matti, thank you for your response. It will be proprietary in the sense that it will belong within the company as part of a training presentation, but we will not be making money from it. These training presentations go into a growing archive of high performance computing, ranging from instructions on how to parallelize code on GPUs to specific tools that will help code run faster (ex. cython, numba and hopefully pypy). The training will not have anything enlightening on it. It's intended as a general introduction to the tool and when to use it (as you know it's slow when calling C extension modules and performing very short, simple tasks compared to python) Just that sort of stuff. Here is what I'm specifically looking for: A high level overview of how pypy works starting from source code up through the end of execution. I'd like something analogous to this description and diagram of cpython: The cpython interpreter (written in C) parses the syntax and creates bytecode. This bytecode is then executed by the python virtual machine (also written in C). [cid:c0457197-8bba-4e57-a906-d3b9e5412dcb] So I'm not going into details like garbage collection, the GIL, or even details of the comiler. Similarly I'd like something that doesn't go too deeply into the innerworkings of pypy, but gives a good idea of how the code is handled at each stage during the execution. I've found several diagrams and charts but they don't seem to include the whole picture or are too detailed and I can't seem to make sense of it. Here are some diagrams or flowcharts I've found: [cid:c3777ff1-e1e0-4f1c-bce9-c709d93401df][cid:0fc18619-9475-4f5d-8d8c-3256276825b2][cid:5e604678-8ce5-4031-94fd-0df620bfbd77][cid:7aa29236-d0a1-4ab8-8d47-18f683931ad4] For pypy, I'm uncertain for example if source code is converted into analogous rpython then that's converted to C, or if it's converted to bytecode then C. I'm not certain where the jit kicks in, I know it's provided by rpython and it kicks in on hot (>1031 loops). If a guard fails, I'm uncertain where the code returns to being interpreted. I feel it would be quicker if we could chat on zoom or other platform to share screens, it may be the fastest. I feel a good high level overview would be a good blog post for pypy or be included in the introductory documentation. Trevor Clack HPCMP PET Computational Scientist Mobile 949.412.9902 Trevor.Clack at gdit.com 3909 Halls Ferry Road Vicksburg, MS 39180 www.gdit.com [cid:image001.png at 01D664F5.C0E0A040] ________________________________ From: Matti Picus Sent: Wednesday, September 30, 2020 12:42 AM To: Clack, Trevor ; pypy-dev at python.org Subject: Re: [pypy-dev] Short discussion about pypy overview [External: Use caution with links & attachments] On 9/30/20 2:15 AM, Clack, Trevor wrote: > Hello, I'm working on giving a presentation to a high performance > computing community on pypy. I've got a good deal of content developed > except for a high level overview of how pypy in fact works. Several > days of YouTube, searching stack overflow, and searching the docs > haven't clarified much. I was hoping to get in contact with a > developer if it's not too much trouble > > Trevor Clack > > HPCMP PET Computational Scientist > > Mobile 949.412.9902 > > Trevor.Clack at gdit.com > > 3909 Halls Ferry Road > > Vicksburg, MS 39180 > > www.gdit.com > > cid:image001.png at 01D664F5.C0E0A040 > > Hi and welcome. Could you be a little more specific: do you mean "how the RPython tool chain can compile a Python interpreter", "how is PyPy fast?", "what optimizations can the JIT do", "how does a tracing JIT work, "why doesn't PyPy use refcount semantics", or something else? Matti -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cpython_flowchart.png Type: image/png Size: 24067 bytes Desc: cpython_flowchart.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart1.png Type: image/png Size: 54738 bytes Desc: pypy_chart1.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart2.png Type: image/png Size: 22546 bytes Desc: pypy_chart2.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart3.png Type: image/png Size: 79433 bytes Desc: pypy_chart3.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart4.png Type: image/png Size: 86492 bytes Desc: pypy_chart4.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-cid_image0.png Type: image/png Size: 7680 bytes Desc: Outlook-cid_image0.png URL: From matti.picus at gmail.com Wed Sep 30 13:47:37 2020 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 30 Sep 2020 20:47:37 +0300 Subject: [pypy-dev] Short discussion about pypy overview In-Reply-To: References: Message-ID: <2d132fcf-49fb-9533-e1af-2e48e66f248c@gmail.com> Top posting to avoid too much scrolling. PyPy has two parts that work together: a byte code interpreter like CPython's, and a JIT. The difference is that the interpreter has an additional layer. All functions and loops (for simplicity, let's stick with those) are traced, and the tracer knows the arguments provided to functions and loops. After the tracer sees ~1000 calls to the same code with the same types, it will replace that piece of code with specialized code for those types of arguments. This is typical of a tracing JIT. It will also add guards so that if the assumptions are broken the old, interpreted code, will be run. Thus only specialized code is turned into assembler. Some other JIT frameworks, like Numba, produce LLVM IR code rather than the assembler code that PyPy produces, and then must call a compiler to finish the job. PyPy's RPython knows how to directly emit assembler. I am sure my description is not 100% accurate, others are invited to correct me. We can continue the discussion on ZOOM, would? 11:00 UTC be suitable tomorrow? Matti On 9/30/20 6:52 PM, Clack, Trevor wrote: > Hi Matti, thank you for your response. > > It will be proprietary in the sense that it will belong within the > company as part of a training presentation, but we will not be making > money from it. These training presentations go into a growing archive > of high performance computing, ranging from instructions on how to > parallelize code on GPUs to specific tools that will help code run > faster (ex. cython, numba and hopefully pypy). > > ?The training will not have anything enlightening on it. It's intended > as a general introduction to the tool and when to use it (as you know > it's slow when calling C extension modules and performing very short, > simple tasks compared to python) Just that sort of stuff. > > Here is what I'm specifically looking for: > > A high level overview of how pypy works starting from source code up > through the end of execution. I'd like something analogous to this > description and diagram of cpython: > > The cpython interpreter (written in C) parses the syntax and creates > bytecode. This bytecode is then executed by the python virtual machine > (also written in C). > > > > So I'm not going into details like garbage collection, the GIL, or > even details of the comiler. Similarly I'd like something that doesn't > go too deeply into the innerworkings of pypy, but gives a good idea of > how the code is handled at each stage during the execution. I've found > several diagrams and charts but they don't seem to include the whole > picture or are too detailed and I can't seem to make sense of it. > > Here are some diagrams or flowcharts I've found: > > > For pypy, I'm uncertain for example if source code is converted into > analogous rpython then that's converted to C, or if it's converted to > bytecode then C. I'm not certain where the jit kicks in, I know it's > provided by rpython and it kicks in on hot (>1031 loops). If a guard > fails, I'm uncertain where the code returns to being interpreted. I > feel it would be quicker if we could chat on zoom or other platform to > share screens, it may be the fastest. > > I feel a good high level overview would be a good blog post for pypy > or be included in the introductory documentation. > > Trevor Clack > > HPCMP PET Computational Scientist > > Mobile 949.412.9902 > > Trevor.Clack at gdit.com > > 3909 Halls Ferry Road > > Vicksburg, MS 39180 > > www.gdit.com > > cid:image001.png at 01D664F5.C0E0A040 > > From matti.picus at gmail.com Wed Sep 30 18:01:09 2020 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 1 Oct 2020 01:01:09 +0300 Subject: [pypy-dev] Short discussion about pypy overview In-Reply-To: References: Message-ID: <1a1c18d6-79ca-242a-e300-eedd595fc64f@gmail.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cpython_flowchart.png Type: image/png Size: 24067 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart4.png Type: image/png Size: 86492 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart3.png Type: image/png Size: 79433 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart2.png Type: image/png Size: 22546 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_chart1.png Type: image/png Size: 54738 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-cid_image0.png Type: image/png Size: 7680 bytes Desc: not available URL: From ulubas9090 at gmail.com Wed Sep 30 17:12:38 2020 From: ulubas9090 at gmail.com (Mustafa Ulubas) Date: Wed, 30 Sep 2020 22:12:38 +0100 Subject: [pypy-dev] About PySide2 Message-ID: Hi there, Hope you are doing well. I'm currently working on a project which I'm trying to use PyPy in order to speed up our simulation processes. There are a couple of dependencies that our project has including PySide2. I've tried to install PySide2 into PyPy however I couldn't. From what I understand as a result of my investigations, currently, PyPy doesn't support PySide2. First of all, is that true? Does PyPy really not support it? Is there any way to install it? I'll be waiting for your response. Thanks in advance. Mustafa -------------- next part -------------- An HTML attachment was scrubbed... URL: