From andy at delib.net Fri Nov 5 12:59:20 2021 From: andy at delib.net (Andy) Date: Fri, 5 Nov 2021 16:59:20 +0000 Subject: [pypy-dev] M1 support? Message-ID: <10AE17C4-3F29-4BCC-B4D1-0ED13AC04412@andythenorth.co.uk> Hi PyPy First, thanks for the project. It's a real aid to me in a hobby project (game mods), which is compile-time sensitive (pypy3 is much faster here than cpython). Second, per https://www.pypy.org/posts/2020/12/mac-meets-arm64-940822335619099039.html Did you get any contributions towards M1 support? I would not be able to cover the full cost, but I can make a modest contribution, either directly or via Open Collective. cheers, Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: From teammember0x01 at gmail.com Fri Nov 5 16:03:21 2021 From: teammember0x01 at gmail.com (M A) Date: Fri, 5 Nov 2021 16:03:21 -0400 Subject: [pypy-dev] Where is translatorshell.py? Message-ID: <0815453D-52B5-4A60-8149-F3E0E624F661@gmail.com> I was reading the information from https://rpython.readthedocs.io/en/latest/translation.html. It talks about a file called bin/translatorshell.py. I could not locate the file. Would anyone know where this file is located? Thank you. From teammember0x01 at gmail.com Sat Nov 6 12:15:59 2021 From: teammember0x01 at gmail.com (M A) Date: Sat, 6 Nov 2021 12:15:59 -0400 Subject: [pypy-dev] M1 support? In-Reply-To: References: Message-ID: Hello Andy, I am currently working on M1 support. I keep my patches here: https://foss.heptapod.net/pypy/pypy/-/issues/3566. I was able to build an ARM64 binary of PyPy for M1 that did run a small prime number generator program I made. Unfortunately PyPy still crashes a lot. I did talk to Armin Rigo about porting PyPy to M1 Macs. The only solution I see for him is to buy an M1 MacMini. Emulation of an M1 Mac is probably a decade away ? - so that is not an option. In the page you site there were people who volunteered ssh login support to their computers. If someone was willing to leave their computer on 24/7 then that option might work also. Thank you. > On Nov 6, 2021, at 12:00 PM, pypy-dev-request at python.org wrote: > > Hi PyPy > > First, thanks for the project. It's a real aid to me in a hobby project (game mods), which is compile-time sensitive (pypy3 is much faster here than cpython). > > Second, per https://www.pypy.org/posts/2020/12/mac-meets-arm64-940822335619099039.html > > Did you get any contributions towards M1 support? I would not be able to cover the full cost, but I can make a modest contribution, either directly or via Open Collective. > > cheers, > > Andy From georges.racinet at octobus.net Sat Nov 6 12:51:13 2021 From: georges.racinet at octobus.net (Georges Racinet) Date: Sat, 6 Nov 2021 17:51:13 +0100 Subject: [pypy-dev] M1 support? In-Reply-To: References: Message-ID: Hi there, just in case, the GCC compile farm has an available M1, see https://cfarm.tetaneutral.net/news/30# This is a modest machine, with a small disk, but perhaps it can be useful ? I suppose that PyPy would meet the requirements. I've used it to compile Heptapod Runner once or twice. Best, On 11/6/21 5:15 PM, M A wrote: > Hello Andy, > > I am currently working on M1 support. I keep my patches here: https://foss.heptapod.net/pypy/pypy/-/issues/3566. I was able to build an ARM64 binary of PyPy for M1 that did run a small prime number generator program I made. Unfortunately PyPy still crashes a lot. > > I did talk to Armin Rigo about porting PyPy to M1 Macs. The only solution I see for him is to buy an M1 MacMini. Emulation of an M1 Mac is probably a decade away ? - so that is not an option. In the page you site there were people who volunteered ssh login support to their computers. If someone was willing to leave their computer on 24/7 then that option might work also. > > Thank you. > > >> On Nov 6, 2021, at 12:00 PM, pypy-dev-request at python.org wrote: >> >> Hi PyPy >> >> First, thanks for the project. It's a real aid to me in a hobby project (game mods), which is compile-time sensitive (pypy3 is much faster here than cpython). >> >> Second, per https://www.pypy.org/posts/2020/12/mac-meets-arm64-940822335619099039.html >> >> Did you get any contributions towards M1 support? I would not be able to cover the full cost, but I can make a modest contribution, either directly or via Open Collective. >> >> cheers, >> >> Andy > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev -- Georges Racinet https://octobus.net, https://about.heptapod.host, https://heptapod.net GPG: BF5456F4DC625443849B6E58EE20CA44EF691D39, sur serveurs publics From cfbolz at gmx.de Sun Nov 7 14:40:28 2021 From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick) Date: Sun, 7 Nov 2021 20:40:28 +0100 Subject: [pypy-dev] plans for the peg parser in pypy 3.9 Message-ID: <3d8c71ad-ad13-7e4f-0286-b59c7307ad8f@gmx.de> Hi all, so I've been hacking on getting an rpython version of the new peg parser that cpython introduced in 3.9, and switched exclusively to in 3.10. My status on the py3.9-peg-parser branch is that I managed to build a full pypy-c with the new parser for the first time today, but there are still a number of bugs (in particular, the interactive console is pretty broken). I wanted to keep you in the loop on my plans for further steps: - I plan to fix the bugs, obviously ;-) - CPython made it possible to switch between the old and the new parser in 3.9 with an environment variable. Unless somebody gives extremely strong reasons for also having that feature, I would like to focus on using the new parser exclusively. Being able to switch is quite a big hassle, and I don't see it buying a lot. - That said, I intend to keep the "parser" module functional for 3.9, like CPython did. It will be removed (for us and for them) in 3.10. Let me know if you are unhappy with that approach! Cheers, Carl Friedrich From hodgestar at gmail.com Sun Nov 7 19:04:01 2021 From: hodgestar at gmail.com (Simon Cross) Date: Mon, 8 Nov 2021 01:04:01 +0100 Subject: [pypy-dev] plans for the peg parser in pypy 3.9 In-Reply-To: <3d8c71ad-ad13-7e4f-0286-b59c7307ad8f@gmx.de> References: <3d8c71ad-ad13-7e4f-0286-b59c7307ad8f@gmx.de> Message-ID: > - CPython made it possible to switch between the old and the new parser > in 3.9 with an environment variable. Unless somebody gives extremely > strong reasons for also having that feature, I would like to focus on > using the new parser exclusively. Being able to switch is quite a big > hassle, and I don't see it buying a lot. That approach sounds good to me! From teammember0x01 at gmail.com Mon Nov 8 14:41:35 2021 From: teammember0x01 at gmail.com (M A) Date: Mon, 8 Nov 2021 14:41:35 -0500 Subject: [pypy-dev] How to make a debug build of PyPy Message-ID: <25398A80-D220-48C6-9149-3BB0AB2E9923@gmail.com> Hi, I need to build PyPy with debug symbols in it. The documentation I found online says to use 'make lldebug' or 'make lldebug0'. This did not work. I am building the JIT version of PyPy. Any help or suggestions would be appreciated. Thank you. From cfbolz at gmx.de Mon Nov 8 16:13:44 2021 From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick) Date: Mon, 8 Nov 2021 22:13:44 +0100 Subject: [pypy-dev] How to make a debug build of PyPy In-Reply-To: <25398A80-D220-48C6-9149-3BB0AB2E9923@gmail.com> References: <25398A80-D220-48C6-9149-3BB0AB2E9923@gmail.com> Message-ID: this should work: rpython -Ojit --lldebug targetpypystandalone.py Cheers, CF On 08.11.21 20:41, M A wrote: > Hi, I need to build PyPy with debug symbols in it. The documentation I found online says to use 'make lldebug' or 'make lldebug0'. This did not work. I am building the JIT version of PyPy. Any help or suggestions would be appreciated. Thank you. From cfbolz at gmx.de Tue Nov 9 12:29:10 2021 From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick) Date: Tue, 9 Nov 2021 18:29:10 +0100 Subject: [pypy-dev] plans for the peg parser in pypy 3.9 In-Reply-To: <3d8c71ad-ad13-7e4f-0286-b59c7307ad8f@gmx.de> References: <3d8c71ad-ad13-7e4f-0286-b59c7307ad8f@gmx.de> Message-ID: <5aff20b3-cd2d-37d8-0c78-5ea448d5422e@gmx.de> On 07.11.21 20:40, Carl Friedrich Bolz-Tereick wrote: > - That said, I intend to keep the "parser" module functional for 3.9, > like CPython did. It will be removed (for us and for them) in 3.10. It seems that won't completely work. the module makes it possible to compile a concrete syntax tree to bytecode, which we can't do without keeping the old astbuilder around (which I don't want to do). So the .compile method on stnodes will be disabled. Cheers, Carl Friedrich From teammember0x01 at gmail.com Tue Nov 9 19:49:01 2021 From: teammember0x01 at gmail.com (M A) Date: Tue, 9 Nov 2021 19:49:01 -0500 Subject: [pypy-dev] How to make a debug build of PyPy In-Reply-To: References: Message-ID: <927B897F-8603-417F-8409-1B8846A464F9@gmail.com> Thank you so much. It worked. > On Nov 9, 2021, at 12:00 PM, pypy-dev-request at python.org wrote: > > Message: 2 > Date: Mon, 8 Nov 2021 22:13:44 +0100 > From: Carl Friedrich Bolz-Tereick > To: pypy-dev at python.org > Subject: Re: [pypy-dev] How to make a debug build of PyPy > Message-ID: > Content-Type: text/plain; charset=UTF-8; format=flowed > > this should work: > > rpython -Ojit --lldebug targetpypystandalone.py > > Cheers, > > CF > > On 08.11.21 20:41, M A wrote: >> Hi, I need to build PyPy with debug symbols in it. The documentation I found online says to use 'make lldebug' or 'make lldebug0'. This did not work. I am building the JIT version of PyPy. Any help or suggestions would be appreciated. Thank you. From omer.drow at gmail.com Wed Nov 10 10:53:07 2021 From: omer.drow at gmail.com (Omer Katz) Date: Wed, 10 Nov 2021 17:53:07 +0200 Subject: [pypy-dev] plans for the peg parser in pypy 3.9 In-Reply-To: <5aff20b3-cd2d-37d8-0c78-5ea448d5422e@gmx.de> References: <3d8c71ad-ad13-7e4f-0286-b59c7307ad8f@gmx.de> <5aff20b3-cd2d-37d8-0c78-5ea448d5422e@gmx.de> Message-ID: Kudos on the great work! I honestly don't think there's a need to keep the old parser around if CPython already proved that the new one works just fine. I think we're allowed to be less conservative here because of that. We can always revert your patch if something goes wrong. ??????? ??? ??, 9 ????? 2021 ?-19:29 ??? ?Carl Friedrich Bolz-Tereick?? :? > On 07.11.21 20:40, Carl Friedrich Bolz-Tereick wrote: > > - That said, I intend to keep the "parser" module functional for 3.9, > > like CPython did. It will be removed (for us and for them) in 3.10. > > It seems that won't completely work. the module makes it possible to > compile a concrete syntax tree to bytecode, which we can't do without > keeping the old astbuilder around (which I don't want to do). So the > .compile method on stnodes will be disabled. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at vazor.com Thu Nov 11 17:36:11 2021 From: matt at vazor.com (Matt Billenstein) Date: Thu, 11 Nov 2021 22:36:11 +0000 Subject: [pypy-dev] macos buildbot offline for a few days Message-ID: <0101017d1124e7dc-4fa25d30-eea2-42e8-9e00-77d0a7d53f66-000000@us-west-2.amazonses.com> Starting tomorrow morning PST, I'm moving, so it should be back sometime Saturday evening or perhaps Monday if I have trouble with the internet self-install. thx m -- Matt Billenstein matt at vazor.com http://www.vazor.com/ From matti.picus at gmail.com Fri Nov 12 01:17:39 2021 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 12 Nov 2021 08:17:39 +0200 Subject: [pypy-dev] macos buildbot offline for a few days In-Reply-To: <0101017d1124e7dc-4fa25d30-eea2-42e8-9e00-77d0a7d53f66-000000@us-west-2.amazonses.com> References: <0101017d1124e7dc-4fa25d30-eea2-42e8-9e00-77d0a7d53f66-000000@us-west-2.amazonses.com> Message-ID: On 12/11/21 12:36 am, Matt Billenstein via pypy-dev wrote: > Starting tomorrow morning PST, I'm moving, so it should be back sometime > Saturday evening or perhaps Monday if I have trouble with the internet > self-install. > > thx > > m > Thanks for the heads up and thanks for allowing us to use your machine. I hope the transition goes smoothly. Matti From teammember0x01 at gmail.com Sun Nov 14 15:11:38 2021 From: teammember0x01 at gmail.com (M A) Date: Sun, 14 Nov 2021 15:11:38 -0500 Subject: [pypy-dev] Debugging symbols for JIT compiled code? Message-ID: <530E64C4-0235-47BC-B06C-07820E6AFF5D@gmail.com> Is it possible for PyPy to add debugging symbols to the code it creates and writes to memory? I'm trying to figure out why my python program keeps crashing. If I had debugging symbols available I could instantly figure out what function is being called. Thank you. From teammember0x01 at gmail.com Mon Nov 15 06:41:04 2021 From: teammember0x01 at gmail.com (M A) Date: Mon, 15 Nov 2021 06:41:04 -0500 Subject: [pypy-dev] Identify this function Message-ID: for i in range(10000): print(i) Hi, when I run this loop PyPy crashes at around 8800. Using a debugger I found out that this is the AARCH64 assembly function that has the crash. The arrow points to where the crash happens. I believe the crash happens one instruction before the arrow, at blr x17. X17 was set to a small number that was not a valid address. Would anyone know what the rpython version of this function is? Thank you. 0x10a97cf10: str x0, [x29, #0x48] 0x10a97cf14: str x1, [x29, #0x50] 0x10a97cf18: str x2, [x29, #0x58] 0x10a97cf1c: str x3, [x29, #0x60] 0x10a97cf20: str x4, [x29, #0x68] 0x10a97cf24: str x5, [x29, #0x70] 0x10a97cf28: str x6, [x29, #0x78] 0x10a97cf2c: str x7, [x29, #0x80] 0x10a97cf30: str x8, [x29, #0x88] 0x10a97cf34: str x9, [x29, #0x90] 0x10a97cf38: str x10, [x29, #0x98] 0x10a97cf3c: str x11, [x29, #0xa0] 0x10a97cf40: str x12, [x29, #0xa8] 0x10a97cf44: str x13, [x29, #0xb0] 0x10a97cf48: str x19, [x29, #0xb8] 0x10a97cf4c: str x20, [x29, #0xc0] 0x10a97cf50: str d0, [x29, #0xc8] 0x10a97cf54: str d1, [x29, #0xd0] 0x10a97cf58: str d2, [x29, #0xd8] 0x10a97cf5c: str d3, [x29, #0xe0] 0x10a97cf60: str d4, [x29, #0xe8] 0x10a97cf64: str d5, [x29, #0xf0] 0x10a97cf68: str d6, [x29, #0xf8] 0x10a97cf6c: str d7, [x29, #0x100] 0x10a97cf70: sub sp, sp, #0x10 ; =0x10 0x10a97cf74: str x16, [sp, #0x8] 0x10a97cf78: str x30, [sp] 0x10a97cf7c: blr x17 -> 0x10a97cf80: mov x16, #0x3428 0x10a97cf84: movk x16, #0x69f, lsl #16 0x10a97cf88: movk x16, #0x1, lsl #32 0x10a97cf8c: ldr x16, [x16] 0x10a97cf90: sub x16, x16, #0x8 ; =0x8 0x10a97cf94: ldr x29, [x16] 0x10a97cf98: ldrb w16, [x29, #0x4] 0x10a97cf9c: mov x17, #0x1 0x10a97cfa0: tst x16, x17 0x10a97cfa4: b.eq 0x10a97cfd8 0x10a97cfa8: sub sp, sp, #0x10 ; =0x10 0x10a97cfac: str x0, [sp, #0x8] 0x10a97cfb0: str x29, [sp] 0x10a97cfb4: mov x0, x29 0x10a97cfb8: mov x16, #0xc290 0x10a97cfbc: movk x16, #0xa97, lsl #16 0x10a97cfc0: movk x16, #0x1, lsl #32 0x10a97cfc4: movk x16, #0x0, lsl #48 0x10a97cfc8: blr x16 0x10a97cfcc: ldr x0, [sp, #0x8] 0x10a97cfd0: ldr x29, [sp] 0x10a97cfd4: add sp, sp, #0x10 ; =0x10 0x10a97cfd8: mov x17, x0 0x10a97cfdc: ldr x0, [x29, #0x48] 0x10a97cfe0: ldr x1, [x29, #0x50] 0x10a97cfe4: ldr x2, [x29, #0x58] 0x10a97cfe8: ldr x3, [x29, #0x60] 0x10a97cfec: ldr x4, [x29, #0x68] 0x10a97cff0: ldr x5, [x29, #0x70] 0x10a97cff4: ldr x6, [x29, #0x78] 0x10a97cff8: ldr x7, [x29, #0x80] 0x10a97cffc: ldr x8, [x29, #0x88] 0x10a97d000: ldr x9, [x29, #0x90] 0x10a97d004: ldr x10, [x29, #0x98] 0x10a97d008: ldr x11, [x29, #0xa0] 0x10a97d00c: ldr x12, [x29, #0xa8] 0x10a97d010: ldr x13, [x29, #0xb0] 0x10a97d014: ldr x19, [x29, #0xb8] 0x10a97d018: ldr x20, [x29, #0xc0] 0x10a97d01c: ldr x16, [sp] 0x10a97d020: add sp, sp, #0x10 ; =0x10 0x10a97d024: ret x16 From teammember0x01 at gmail.com Sun Nov 21 14:16:51 2021 From: teammember0x01 at gmail.com (M A) Date: Sun, 21 Nov 2021 14:16:51 -0500 Subject: [pypy-dev] How to compile a file with rpython Message-ID: <30C07618-205D-4CDE-AAA8-54FD86F0388B@gmail.com> How do I make a program using rpython? Say I have this file: def main(): for i in range(100): print("hello world"), if __name__ == "__main__": main() How would I turn it into an executable file using rpython? From matti.picus at gmail.com Sun Nov 21 16:38:31 2021 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 21 Nov 2021 23:38:31 +0200 Subject: [pypy-dev] How to compile a file with rpython In-Reply-To: <30C07618-205D-4CDE-AAA8-54FD86F0388B@gmail.com> References: <30C07618-205D-4CDE-AAA8-54FD86F0388B@gmail.com> Message-ID: <8eab8d6f-a6b9-7301-9e9b-4da2ad1faaf4@gmail.com> On 21/11/21 9:16 pm, M A wrote: > How do I make a program using rpython? > > Say I have this file: > > def main(): > for i in range(100): > print("hello world"), > > if __name__ == "__main__": > main() > > How would I turn it into an executable file using rpython? The canonical starting place for rpython is this page [0] in the rpython documentation. It is a bit old, so please raise an issue if something does not work right. There are some examples in rpython/translator/goal/, the targetnopstandalone.py one is close to what you are trying to do. If you call python2 rpython/bin/rpython -O2 rpython/translator/goal/targetnopstandalone.py you will get an executable in $tmp/usession*/testing_1/targetnopstandalone-c that will print "hello world" Matti [0] https://rpython.readthedocs.io/en/latest/getting-started.html From teammember0x01 at gmail.com Mon Nov 22 12:06:30 2021 From: teammember0x01 at gmail.com (M A) Date: Mon, 22 Nov 2021 12:06:30 -0500 Subject: [pypy-dev] How to compile a file with rpython In-Reply-To: References: <30C07618-205D-4CDE-AAA8-54FD86F0388B@gmail.com> Message-ID: <9C7DB6F2-56FB-4666-818F-09D384905195@gmail.com> > On Nov 22, 2021, at 1:08 AM, Dan Stromberg wrote: > > > On Sun, Nov 21, 2021 at 11:17 AM M A wrote: > How do I make a program using rpython? > > Say I have this file: > > def main(): > for i in range(100): > print("hello world"), > > if __name__ == "__main__": > main() > > How would I turn it into an executable file using rpython? > > Why do you desire an executable? > > Last I heard, a program would perform well on Pypy's JIT, and that using rpython was almost always unnecessary. > > If you need an executable, you could also try Cython or Nuitka. Well I'm working on code for PyPy. If I could unit test my code my development speed would increase a lot. It takes a long time for PyPy to build on my computer. From matti.picus at gmail.com Mon Nov 22 12:26:21 2021 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 22 Nov 2021 19:26:21 +0200 Subject: [pypy-dev] How to compile a file with rpython In-Reply-To: <9C7DB6F2-56FB-4666-818F-09D384905195@gmail.com> References: <30C07618-205D-4CDE-AAA8-54FD86F0388B@gmail.com> <9C7DB6F2-56FB-4666-818F-09D384905195@gmail.com> Message-ID: <84b079c2-8d6a-c059-2034-ee0cc9e61679@gmail.com> On 22/11/21 7:06 pm, M A wrote: > >> On Nov 22, 2021, at 1:08 AM, Dan Stromberg wrote: >> >> >> Why do you desire an executable? >> >> Last I heard, a program would perform well on Pypy's JIT, and that using rpython was almost always unnecessary. >> >> If you need an executable, you could also try Cython or Nuitka. > Well I'm working on code for PyPy. If I could unit test my code my development speed would increase a lot. It takes a long time for PyPy to build on my computer. > _______________________________________________ The way to unit test code is with python2 pytest.py rpython/path/to/test/test_file.py This is explained in the testing section of the "how to contribute" [0]. If that document is not clear, please help us improve it. Matti [0] https://doc.pypy.org/en/latest/contributing.html#testing From haow85 at live.com Sun Nov 28 23:01:59 2021 From: haow85 at live.com (Hao Wang) Date: Mon, 29 Nov 2021 04:01:59 +0000 Subject: [pypy-dev] pip installation for pypy is slow Message-ID: Dear pypy-devs, The pip installation of pypy (pypy -m pip install scipy, for example) is slower than python-pip clean. Is there a way to fix this problem ? Bravo! Hao Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: From haow85 at live.com Mon Nov 29 00:38:53 2021 From: haow85 at live.com (Hao Wang) Date: Mon, 29 Nov 2021 05:38:53 +0000 Subject: [pypy-dev] How to fix pypy3 runtime error ? Message-ID: Dear devs: I encountered the following error but didn't know how to fix it. Please help. cpyext, the emulation layer, detected that while it is calling an object's tp_dealloc, the C code calls back a function that tries to recreate the PyPy version of the object. Usually it means that tp_dealloc calls some general PyXxx() API. It is a dangerous and potentially buggy thing to do: even in CPython the PyXxx() function could, in theory, cause a reference to the object to be taken and stored somewhere, for an amount of time exceeding tp_dealloc itself. Afterwards, the object will be freed, making that reference point to garbage. >>> PyPy could contain some workaround to still work if you are lucky, but it is not done so far; better fix the bug in the CPython extension. >>> This object is of type 'pandas._libs.parsers.TextReader' Aborted Bravo ! Hao Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Mon Nov 29 01:31:10 2021 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 29 Nov 2021 08:31:10 +0200 Subject: [pypy-dev] pip installation for pypy is slow In-Reply-To: References: Message-ID: On 29/11/21 6:01 am, Hao Wang wrote: > Dear pypy-devs, > > The pip installation of pypy (pypy -m pip install scipy, for example) > is slower than python-pip clean. Is there a way to fix this problem ? > > Bravo! > Hao Wang > Thanks for giving PyPy a try. When pre-compiled binary wheels are not available, python must compile from source. This takes time. We have been making an effort to convince project maintainers to provide binary wheels for PyPy. SciPy has a complicated workflow and a skeleton support team. That together with the fact that currently PyPy + SciPy is much slower to run CI (small snippets of code run in tests are not PyPy's strong point, and the c-api emulation layer makes things worse) mean that SciPy is hesitant to add support for PyPy, see the PR where the CI runs were removed [0] in 2019. Our long-term proposal to solve this problem is to get pybind11 (for c++ code), cython (for c code), and f2py (for fortran code) to generate wrappers using HPy [1] instead of the C-API, which will speed up testing with PyPy.? A second possibility is that the scientific python community will move away from its heavy dependency on the CPython C-API and more towards using higher-level protocols [4]. Both of these efforts will take many years unless heavily sponsored. A more specific short-term solution for your installation problem is to use conda-forge's miniforge [3]. This will solve the installation problem, but PyPy will still be slow on scipy-heavy workflows. The conda-forge project supplies binary packages for pypy3.7, and is discussing pypy3.8 support [2]: conda create -n pypy3.7 pypy conda activate pypy3.7 conda install scipy Matti [0] https://github.com/scipy/scipy/pull/10085 [1] https://hpyproject.org/ [2] https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/2089 [3] https://github.com/conda-forge/miniforge [4] https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision/ From haow85 at live.com Mon Nov 29 01:41:03 2021 From: haow85 at live.com (Hao Wang) Date: Mon, 29 Nov 2021 06:41:03 +0000 Subject: [pypy-dev] pip installation for pypy is slow In-Reply-To: References: Message-ID: Thanks Matti, I switched to Cython due to PyPy's incompatibility with pandas (The error is given below. If you know how to fix this, please let me know.) cpyext, the emulation layer, detected that while it is calling an object's tp_dealloc, the C code calls back a function that tries to recreate the PyPy version of the object. Usually it means that tp_dealloc calls some general PyXxx() API. It is a dangerous and potentially buggy thing to do: even in CPython the PyXxx() function could, in theory, cause a reference to the object to be taken and stored somewhere, for an amount of time exceeding tp_dealloc itself. Afterwards, the object will be freed, making that reference point to garbage. >>> PyPy could contain some workaround to still work if you are lucky, but it is not done so far; better fix the bug in the CPython extension. >>> This object is of type 'pandas._libs.parsers.TextReader' Aborted ________________________________ From: pypy-dev on behalf of Matti Picus Sent: Monday, November 29, 2021 12:31 AM To: pypy-dev at python.org Subject: Re: [pypy-dev] pip installation for pypy is slow On 29/11/21 6:01 am, Hao Wang wrote: > Dear pypy-devs, > > The pip installation of pypy (pypy -m pip install scipy, for example) > is slower than python-pip clean. Is there a way to fix this problem ? > > Bravo! > Hao Wang > Thanks for giving PyPy a try. When pre-compiled binary wheels are not available, python must compile from source. This takes time. We have been making an effort to convince project maintainers to provide binary wheels for PyPy. SciPy has a complicated workflow and a skeleton support team. That together with the fact that currently PyPy + SciPy is much slower to run CI (small snippets of code run in tests are not PyPy's strong point, and the c-api emulation layer makes things worse) mean that SciPy is hesitant to add support for PyPy, see the PR where the CI runs were removed [0] in 2019. Our long-term proposal to solve this problem is to get pybind11 (for c++ code), cython (for c code), and f2py (for fortran code) to generate wrappers using HPy [1] instead of the C-API, which will speed up testing with PyPy. A second possibility is that the scientific python community will move away from its heavy dependency on the CPython C-API and more towards using higher-level protocols [4]. Both of these efforts will take many years unless heavily sponsored. A more specific short-term solution for your installation problem is to use conda-forge's miniforge [3]. This will solve the installation problem, but PyPy will still be slow on scipy-heavy workflows. The conda-forge project supplies binary packages for pypy3.7, and is discussing pypy3.8 support [2]: conda create -n pypy3.7 pypy conda activate pypy3.7 conda install scipy Matti [0] https://github.com/scipy/scipy/pull/10085 [1] https://hpyproject.org/ [2] https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/2089 [3] https://github.com/conda-forge/miniforge [4] https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision/ _______________________________________________ pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Mon Nov 29 02:04:39 2021 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 29 Nov 2021 09:04:39 +0200 Subject: [pypy-dev] How to fix pypy3 runtime error ? In-Reply-To: References: Message-ID: <1ecde342-b099-9d00-946a-7972f3bd761a@gmail.com> On 29/11/21 7:38 am, Hao Wang wrote: > Dear devs: > > I encountered the following error but didn't know how to fix it. > Please help. > > cpyext, the emulation layer, detected that while it is calling > an object's tp_dealloc, the C code calls back a function that > tries to recreate the PyPy version of the object. ?Usually it > means that tp_dealloc calls some general PyXxx() API. ?It is > a dangerous and potentially buggy thing to do: even in CPython > the PyXxx() function could, in theory, cause a reference to the > object to be taken and stored somewhere, for an amount of time > exceeding tp_dealloc itself. ?Afterwards, the object will be > freed, making that reference point to garbage. > >>> PyPy could contain some workaround to still work if > you are lucky, but it is not done so far; better fix the bug in > the CPython extension. > >>> This object is of type 'pandas._libs.parsers.TextReader' > Aborted > > Bravo ! > Hao Wang You don't show any code so it is difficult to guess what is going on. In general, PyPy is more sensitive to closing resources after use rather than depending on the garbage collector to release them when an object is collected. You could try closing the parser before it goes out of scope. Matti From haow85 at live.com Mon Nov 29 02:09:10 2021 From: haow85 at live.com (Hao Wang) Date: Mon, 29 Nov 2021 07:09:10 +0000 Subject: [pypy-dev] How to fix pypy3 runtime error ? In-Reply-To: <1ecde342-b099-9d00-946a-7972f3bd761a@gmail.com> References: <1ecde342-b099-9d00-946a-7972f3bd761a@gmail.com> Message-ID: I think my code doesn't even get through the following snippet : data_pd = pd.read_csv(input_file) cols = [col_0, col_1, col_2] res_pd = data_pd res_pd = res_pd.drop(cols, axis=1) I'd like to know how to close the parser (or what it means) as well. Bravo ! Hao Wang ________________________________ From: pypy-dev on behalf of Matti Picus Sent: Monday, November 29, 2021 1:04 AM To: pypy-dev at python.org Subject: Re: [pypy-dev] How to fix pypy3 runtime error ? On 29/11/21 7:38 am, Hao Wang wrote: > Dear devs: > > I encountered the following error but didn't know how to fix it. > Please help. > > cpyext, the emulation layer, detected that while it is calling > an object's tp_dealloc, the C code calls back a function that > tries to recreate the PyPy version of the object. Usually it > means that tp_dealloc calls some general PyXxx() API. It is > a dangerous and potentially buggy thing to do: even in CPython > the PyXxx() function could, in theory, cause a reference to the > object to be taken and stored somewhere, for an amount of time > exceeding tp_dealloc itself. Afterwards, the object will be > freed, making that reference point to garbage. > >>> PyPy could contain some workaround to still work if > you are lucky, but it is not done so far; better fix the bug in > the CPython extension. > >>> This object is of type 'pandas._libs.parsers.TextReader' > Aborted > > Bravo ! > Hao Wang You don't show any code so it is difficult to guess what is going on. In general, PyPy is more sensitive to closing resources after use rather than depending on the garbage collector to release them when an object is collected. You could try closing the parser before it goes out of scope. Matti _______________________________________________ pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: