From tbaldridge at gmail.com Tue May 1 18:21:04 2018 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 1 May 2018 16:21:04 -0600 Subject: [pypy-dev] RPython translation error on generated code Message-ID: I have a strange error, I'm generating code for a RPython program and get this error: [translation:ERROR] AttributeError: 'FrozenDesc' object has no attribute 'pycall' Processing block: block at 9[fn_0...] is a in (?:10)make_thunk containing the following operations: v2 = simple_call((type Thunk_thunk10_), fn_0, args_0) And I'm generating it thusly: thunk_template = """ class Thunk_{sym}(ThunkBase): def __init__(self, fn, args): self._{sym}_fn = fn self._{sym}_args = args def run_thunk(self): return self._{sym}_fn(*self._{sym}_args) def make_thunk(fn, args): return Thunk_{sym}(fn, args) """ def make_thunk(): gbls = dict(ThunkBase=ThunkBase, object=object) sym = gen_sym.get_sym("thunk") code = thunk_template.format(sym=sym) exec(code, gbls) cls = gbls["make_thunk"] return cls It's as if RPython isn't recognizing that the type is callable. Any thoughts? Thanks, Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbaldridge at gmail.com Tue May 1 18:54:25 2018 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 1 May 2018 16:54:25 -0600 Subject: [pypy-dev] RPython translation error on generated code In-Reply-To: References: Message-ID: Well, I switched to using sourcetools.compile_template and the error went away, so I was probably doing something wrong with how I compiled the template. Timothy On Tue, May 1, 2018 at 4:21 PM, Timothy Baldridge wrote: > I have a strange error, I'm generating code for a RPython program and get > this error: > > [translation:ERROR] AttributeError: 'FrozenDesc' object has no attribute > 'pycall' > Processing block: > block at 9[fn_0...] is a > in (?:10)make_thunk > containing the following operations: > v2 = simple_call((type Thunk_thunk10_), fn_0, args_0) > > > And I'm generating it thusly: > > thunk_template = """ > class Thunk_{sym}(ThunkBase): > def __init__(self, fn, args): > self._{sym}_fn = fn > self._{sym}_args = args > > def run_thunk(self): > return self._{sym}_fn(*self._{sym}_args) > > def make_thunk(fn, args): > return Thunk_{sym}(fn, args) > """ > > > > def make_thunk(): > gbls = dict(ThunkBase=ThunkBase, object=object) > sym = gen_sym.get_sym("thunk") > code = thunk_template.format(sym=sym) > exec(code, gbls) > cls = gbls["make_thunk"] > return cls > > > It's as if RPython isn't recognizing that the type is callable. > > Any thoughts? > > Thanks, > > Timothy > -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bremner at unb.ca Wed May 2 15:21:23 2018 From: bremner at unb.ca (David Bremner) Date: Wed, 02 May 2018 16:21:23 -0300 Subject: [pypy-dev] PhD Positions in programming language runtimes (Python) In-Reply-To: <87a86bxcga.fsf@tethera.net> References: <87a86bxcga.fsf@tethera.net> Message-ID: <87muxhvp70.fsf@tesseract.cs.unb.ca> David Bremner writes: > 1 Position I: Python on OMR > =========================== > > The candidate will explore the benefits and challenges of using OMR > [1] components (e.g. JIT, GC) in Python [2,3] runtimes. This may > involve extending OMR in novel ways to better support Python, or > vice-versa. If questions like how to balance fast startup time for > scripting applications against good performance for compute-intensive > workloads sound interesting, this might an opportunity for you. > We are still looking for people. Ad blurb below, hopefully once a year or so is not too often. ---------------------------------------------------------------------- The IBM Center For Advanced Studies (Atlantic) [1] has several fully funded positions available for graduate students working on Python related projects. Topics include garbage collection, just-in-time compilation, parallel and distributed computing, programming language interoperability, and sandboxing. Students will have a chance to work on open source projects, including OMR [2]. We have a lively group of professors, students, postdocs, and research assistants working on related problems. Students work closely with both UNB researchers and technical experts from IBM, gaining valuable perspective and experience. 1 Qualifications ???????????????? For positions at the Master's level, UNB requires a Bachelor's degree in computer science or equivalent; at the PhD level we normally require a Master's degree in computer science or a closely related area. These project will require substantial hands-on and experimental work, and being comfortable with Python and C++. Familiarity with Linux and Open Source Software development would be a plus. Exposure to programming language concepts like GC, JIT, and bytecode would be an asset. Several positions would benefit from a background in operating systems, or parallel programming frameworks. 2 Support ????????? These positions are "fully funded"; the exact funding package is subject to finalization but will be consistent with other fully funded students in Computer Science at UNB. You can find some information about expenses on the UNB web site[3]. 3 Contact ????????? Send your resum? and unofficial transcripts by email to Stephen MacKay Please specify whether you are interested in a position at the Masters or PhD level, and any listed topics of particular interest. Footnotes ????????? [1] [http://www.unb.ca/research/casatlantic/] [2] [http://www.eclipse.org/omr/] [3] [http://www.unb.ca/fredericton/studentservices/international/new_international_student/moneymatters.html] -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 658 bytes Desc: not available URL: From wlavrijsen at lbl.gov Thu May 3 01:13:48 2018 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 2 May 2018 22:13:48 -0700 (PDT) Subject: [pypy-dev] module _cppyy disabled in py3.5 release? Message-ID: Hi, in the release-pypy3.5-v6.0.0 branch (and hence in the 3.5 release), module _cppyy was disabled: "_csv", "_pypyjson", "_posixsubprocess", # "_cppyy", "micronumpy" The commit history doesn't say? Anything I should fix? Thanks! Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From matti.picus at gmail.com Thu May 3 05:42:19 2018 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 3 May 2018 12:42:19 +0300 Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Thu May 3 06:12:09 2018 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 3 May 2018 13:12:09 +0300 Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: References:

Message-ID: <9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> On 03/05/18 12:42, Matti Picus wrote: > On 03/05/18 08:13, wlavrijsen at lbl.gov wrote: >> Hi, >> >> in the release-pypy3.5-v6.0.0 branch (and hence in the 3.5 release), >> module >> _cppyy was disabled: >> >> ??? "_csv", "_pypyjson", "_posixsubprocess", # "_cppyy", "micronumpy" >> >> The commit history doesn't say? Anything I should fix? >> >> Thanks! >> >> Best regards, >> ?????????? Wim > That seems to originate from 2012, commit f5eb386cd043, so I don't > think we ever tried py3k which became py3.5 with _cppyy. I will give > it try now. > Matti ... and it fails to translate. Anyone want to fix? Matti From lauri at bemyapp.com Thu May 3 06:14:50 2018 From: lauri at bemyapp.com (Lauri Aarnio) Date: Thu, 3 May 2018 12:14:50 +0200 Subject: [pypy-dev] The Biggest EU Observation Challenge by DigitalGlobe & ESA Message-ID: A call for all Python devs: the Biggest EU Observation Challenge by DigitalGlobe & ESA! *DigitalGlobe has teamed up with the European Space Agency (ESA) in order to organize The Earth Observation Challenge . This online challenge turns massive data into groundbreaking solutions that will serve to support disaster relief in developing areas.Participants will get to Identify & map water resources in developing countries and Identify urban change to help mitigate social and environmental risk using DigitalGlobe Library & ESA Sentinel 2 Data. The prizes include 10k? in cash and a possibility to develop your project with ESA.You can import your own Python data to complement the existing sets and it is possible to use Jupyter Notebook.More about the challenge on the website: http://bit.ly/eochallengedg * -- *Lauri Aarnio* Communications Manager *@* BeMyApp 86 rue de Charonne Paris 75011 +33 66 83 93 12 88 | ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Thu May 3 14:23:10 2018 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 3 May 2018 11:23:10 -0700 (PDT) Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: <9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> References:

<9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> Message-ID: Matti, > ... and it fails to translate. Anyone want to fix? thanks for trying; I have the fixes: an indentation error (which does not show up in p2.7 as there is only one branch; bytes <-> text thingy), and the removal of __cmp__ if p3. Where should I put them? They're safe to go into default, but then _cppyy should be enabled separately at some later point in py3. Thanks! Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From matti.picus at gmail.com Fri May 4 03:36:41 2018 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 4 May 2018 10:36:41 +0300 Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: References:

<9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> Message-ID: <3b16e520-e7fd-a21e-1c09-404f91b4e50a@gmail.com> An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Fri May 4 16:51:38 2018 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 4 May 2018 13:51:38 -0700 (PDT) Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: <3b16e520-e7fd-a21e-1c09-404f91b4e50a@gmail.com> References:

<9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> <3b16e520-e7fd-a21e-1c09-404f91b4e50a@gmail.com> Message-ID: Matti, > Translates and runs. There are two new failing tests, > http://buildbot.pypy.org/summary?branch=py3.5&category=linux64 not following the number two? I see three new failing tests: test_pydoc test_site test_app Only one prints something about _cppyy, but I don't understand why. I can't run any of the tests, however: after starting up, they pretty quickly print: test_pydoc.py::unmodified Then spin for almost 10 minutes and print: Traceback (application-level): File "/home/wlav/aditi/pypy/pypy/tool/pytest/run-script/regrverbose.py", line 4 in from test import regrtest, support File "/home/wlav/aditi/pypy/lib-python/3/test/regrtest.py", line 199 in from test import support File "/home/wlav/aditi/pypy/lib-python/3/test/support/__init__.py", line 16 in import nntplib File "/home/wlav/aditi/pypy/lib-python/3/nntplib.py", line 69 in import datetime ImportError: bad magic number in 'datetime': b'\n\xf3\r\n' FAILED Anyway, my best guess is that somehow some _cppyy code runs at startup. There should be none of that. I'll copy over the failing pydoc test from p3 to p2 and see whether I can at least run tests there. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From cfbolz at gmx.de Sat May 5 05:12:27 2018 From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick) Date: Sat, 05 May 2018 11:12:27 +0200 Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: References:

<9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> <3b16e520-e7fd-a21e-1c09-404f91b4e50a@gmail.com> Message-ID: <53D2F0E8-DB50-40ED-B003-3BE69CACEA15@gmx.de> Hi Wim, How are you running these CPython tests? The best way to run them is using a translated pypy-c and then something like pypy-c test_pydoc.py Cheers, Carl Friedrich On May 4, 2018 10:51:38 PM GMT+02:00, wlavrijsen at lbl.gov wrote: >Matti, > >> Translates and runs. There are two new failing tests, >> http://buildbot.pypy.org/summary?branch=py3.5&category=linux64 > >not following the number two? I see three new failing tests: > > test_pydoc > test_site > test_app > >Only one prints something about _cppyy, but I don't understand why. > >I can't run any of the tests, however: after starting up, they pretty >quickly print: > >test_pydoc.py::unmodified > >Then spin for almost 10 minutes and print: > >Traceback (application-level): >File >"/home/wlav/aditi/pypy/pypy/tool/pytest/run-script/regrverbose.py", >line 4 in > from test import regrtest, support >File "/home/wlav/aditi/pypy/lib-python/3/test/regrtest.py", line 199 in > > from test import support >File "/home/wlav/aditi/pypy/lib-python/3/test/support/__init__.py", >line 16 in > import nntplib >File "/home/wlav/aditi/pypy/lib-python/3/nntplib.py", line 69 in > > import datetime >ImportError: bad magic number in 'datetime': b'\n\xf3\r\n' >FAILED > >Anyway, my best guess is that somehow some _cppyy code runs at startup. >There should be none of that. > >I'll copy over the failing pydoc test from p3 to p2 and see whether I >can >at least run tests there. > >Best regards, > Wim >-- >WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- >www.lavrijsen.net >_______________________________________________ >pypy-dev mailing list >pypy-dev at python.org >https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Mon May 7 00:43:05 2018 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Sun, 6 May 2018 21:43:05 -0700 (PDT) Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: <53D2F0E8-DB50-40ED-B003-3BE69CACEA15@gmx.de> References:

<9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> <3b16e520-e7fd-a21e-1c09-404f91b4e50a@gmail.com> <53D2F0E8-DB50-40ED-B003-3BE69CACEA15@gmx.de> Message-ID: Carl, thanks for your help! > How are you running these CPython tests? Using the pypy/test_all.py script. > The best way to run them is using a translated pypy-c and then something > like > > pypy-c test_pydoc.py That sounds like an expensive debugging cycle? At least on my machine, translation takes a long time. Of course, if an individual test takes more than 10mins, than translating is till preferable. :P But anyway, no joy. I get the exact same error: ../../../pypy/goal/pypy3-c test_pydoc.py Traceback (most recent call last): File "test_pydoc.py", line 15, in import test.support File "/home/wlav/aditi/pypy/lib-python/3/test/support/__init__.py", line 16, in import nntplib File "/home/wlav/aditi/pypy/lib-python/3/nntplib.py", line 69, in import datetime ImportError: bad magic number in 'datetime': b'\n\xf3\r\n' For the heck of it, I threw it into bing. Turns out that it is not a pypy specific problem and wiping all .pyc files fixes it. I can now reproduce the error. The issue is that pydoc.ModuleScanner imports all builtin modules: for modname in sys.builtin_module_names: which then includes _cppyy. I'm going to make the loading of the backend deferred to the first use. That should make importing innocuous. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From wlavrijsen at lbl.gov Tue May 8 00:23:48 2018 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Mon, 7 May 2018 21:23:48 -0700 (PDT) Subject: [pypy-dev] module _cppyy disabled in py3.5 release? In-Reply-To: <53D2F0E8-DB50-40ED-B003-3BE69CACEA15@gmx.de> References:

<9e32b50c-0d29-63a1-5695-788f34b71f39@gmail.com> <3b16e520-e7fd-a21e-1c09-404f91b4e50a@gmail.com> <53D2F0E8-DB50-40ED-B003-3BE69CACEA15@gmx.de> Message-ID: Matti, thanks for your patience. Fix is in default, so next time when default into the py3.5 brach, the error should go away. The fix is to completely defer the loading of the backend shared library to the point of first use. Thus importing _cppyy (and checking its __doc__ or running help() on it) is now safe. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From cfbolz at gmx.de Sat May 12 10:54:36 2018 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sat, 12 May 2018 16:54:36 +0200 Subject: [pypy-dev] py 3.6 bytecode changes, state of py3.6 Message-ID: <02d9001e-44a9-5d94-e952-42ad33e5bfc7@gmx.de> Hi all, Today I found out why on the py3.6 branch dis does not work properly. It's because cpython 3.6 switched to a slightly different bytecode format, where every instruction is 16 bit long, see these issues: https://bugs.python.org/issue26647 https://bugs.python.org/issue28050 Do we want to implement that too, so that we stay closer to what CPython is doing? Or should I partially undo the 3.6 changes to dis.py to fix it? Also, does anybody have a high-level overview which big things are missing on the py3.6 branch? Or what the current blockers are? Cheers, Carl Friedrich From cfbolz at gmx.de Sat May 12 11:03:16 2018 From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick) Date: Sat, 12 May 2018 17:03:16 +0200 Subject: [pypy-dev] EuroScipy Message-ID: Hi all, Somebody on Twitter asked whether there would be a pypy talk at EuroScipy which is from August 28 to September 1 in Trento, Italy. Here is the Call for Presentations: https://pretalx.com/euroscipy18/cfp NB: the deadline is tomorrow! Carl Friedrich -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmueller at python-academy.de Sun May 13 09:40:34 2018 From: mmueller at python-academy.de (=?UTF-8?Q?Mike_M=c3=bcller?=) Date: Sun, 13 May 2018 15:40:34 +0200 Subject: [pypy-dev] EuroScipy In-Reply-To: References: Message-ID: Am 12.05.18 um 17:03 schrieb Carl Friedrich Bolz-Tereick: > Hi all, > > Somebody on Twitter asked whether there would be a pypy talk at EuroScipy which > is from August 28 to September 1 in Trento, Italy. Here is the Call for > Presentations: https://pretalx.com/euroscipy18/cfp > NB: the deadline is tomorrow! There is also the possibility to give a tutorial, if this makes sense and works better for a topic. Mike > > Carl Friedrich > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From oxylibrium at gmail.com Wed May 16 10:31:23 2018 From: oxylibrium at gmail.com (Oxylibrium Person) Date: Wed, 16 May 2018 20:01:23 +0530 Subject: [pypy-dev] More memory allocated for GC than arenas+rawmalloc+nursery (bug?) Message-ID: So, I?ve been testing moving some of my projects to using PyPy as an interpreter from CPython, but PyPy 6.0.0/ Python 3.5.3 seems to memory bloat and trigger OOM protections in about 6 hours on a server with 2GB of RAM, while CPython 3.6.2 ran for 24 days with a more or less constant memory footprint of ~670MB (+/- 40MB) on my discord bot, Birb. Output of gc.get_stats() about 2-3 hours before the crash are as follows: Total memory consumed: GC used: 641.5MB (peak: 775.5MB) in arenas: 536.5MB rawmalloced: 102.9MB nursery: 2.0MB raw assembler used: 5.5MB ?????????????? Total: 647.0MB Total memory allocated: GC allocated: 1235.9MB (peak: 1303.8MB) in arenas: 602.8MB rawmalloced: 170.8MB nursery: 2.0MB raw assembler allocated: 6.0MB ?????????????? Total: 1241.9MB This behaviour can be successfully replicated on brew?s macOS build of PyPy3 6.0.0 and the official PyPy3 6.0.0 release for 64-bit Ubuntu. To replicate this behaviour, I can use the following code, after installing the rewrite branch on discord.py from git+https://github.com/Rapptz/discord.py at rewrite#egg=discord.py; ``` import gc from discord.ext import commands bot = commands.AutoShardedBot(?>>?) # Variable passed is prefix, changing will not affect result @bot.command() async def get_gc_state(ctx): await ctx.send(gc.get_stats()) bot.run(TOKEN) # Replace TOKEN with authentication token, number of guilds/users visible will affect result ``` Unfortunately, I can only reproduce this issue on my production instance, with ~8.6k guilds, ~143.7k channels, and ~305k visible user accounts. On my test instance with 1 guild, 7 channels and 12 users, there are no measurable memory issues. Of note is that under total memory allocated, the sum of the parts of the GC allocation (602.8+170.8+2.0=775.6) is just a little over 62.75% of the total memory reported as being allocated for the GC. Is this a PyPy bug? Thanks for reading, Oxylibrium. From anto.cuni at gmail.com Wed May 16 12:46:03 2018 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 16 May 2018 18:46:03 +0200 Subject: [pypy-dev] EuroScipy In-Reply-To: References: Message-ID: Hi, I am planning to go, since it's close and it's a beautiful place. Did anybody else already submitted a pypy talk? On Sat, May 12, 2018 at 5:03 PM, Carl Friedrich Bolz-Tereick wrote: > Hi all, > > Somebody on Twitter asked whether there would be a pypy talk at EuroScipy > which is from August 28 to September 1 in Trento, Italy. Here is the Call > for Presentations: https://pretalx.com/euroscipy18/cfp > NB: the deadline is tomorrow! > > Carl Friedrich > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed May 16 12:54:45 2018 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 16 May 2018 09:54:45 -0700 Subject: [pypy-dev] EuroScipy In-Reply-To: References: Message-ID: <0b69a8e0-7731-06b0-3d8f-a725482430ec@gmail.com> I submitted a tutorial on cffi, ctypes, cython based on a talk I gave a year ago, and a talk about "working on NumPy full-time" Hopefully the stars will align and we will meet there Matti On 16/05/18 09:46, Antonio Cuni wrote: > Hi, > I am planning to go, since it's close and it's a beautiful place. > > Did anybody else already submitted a pypy talk? > > On Sat, May 12, 2018 at 5:03 PM, Carl Friedrich Bolz-Tereick > > wrote: > > Hi all, > > Somebody on Twitter asked whether there would be a pypy talk at > EuroScipy which is from August 28 to September 1 in Trento, Italy. > Here is the Call for Presentations: > https://pretalx.com/euroscipy18/cfp > > NB: the deadline is tomorrow! > > Carl Friedrich > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From devnexen at gmail.com Fri May 18 10:09:39 2018 From: devnexen at gmail.com (David CARLIER) Date: Fri, 18 May 2018 14:09:39 +0000 Subject: [pypy-dev] [PATCH] FreeBSD build fix Message-ID: Hi dear list, I have hard time to fork the project thus I m sendng this little patch to fix FreeBSD build. Hope it is helpful. Kind regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy.diff Type: text/x-patch Size: 1051 bytes Desc: not available URL: From vincent.legoll at gmail.com Fri May 18 10:27:25 2018 From: vincent.legoll at gmail.com (Vincent Legoll) Date: Fri, 18 May 2018 16:27:25 +0200 Subject: [pypy-dev] [PATCH] FreeBSD build fix In-Reply-To: References: Message-ID: Hello, Disclaimer: I'm not sure I know what I'm talking about, I'm not really a pypy dev, so take my below questions with that in mind. They may be completely wrong. On Fri, May 18, 2018 at 4:09 PM, David CARLIER wrote: > Hi dear list, I have hard time to fork the project thus I m sendng this > little patch to fix FreeBSD build. Are you sure your startswith('amd64') should not be a == 'amd64' Looks like you enable more than just "freebsd-*/amd64" with this change, are the other combinations tested / working ? Thanks -- Vincent Legoll From devnexen at gmail.com Fri May 18 10:29:56 2018 From: devnexen at gmail.com (David CARLIER) Date: Fri, 18 May 2018 14:29:56 +0000 Subject: [pypy-dev] [PATCH] FreeBSD build fix In-Reply-To: References: Message-ID: Hi and thanks for the feedback; indeed == amd64 is the good expression. On 18 May 2018 at 14:27, Vincent Legoll wrote: > Hello, > > Disclaimer: I'm not sure I know what I'm talking about, I'm not really > a pypy dev, > so take my below questions with that in mind. They may be completely wrong. > > On Fri, May 18, 2018 at 4:09 PM, David CARLIER wrote: > > Hi dear list, I have hard time to fork the project thus I m sendng this > > little patch to fix FreeBSD build. > > Are you sure your startswith('amd64') should not be a == 'amd64' > > Looks like you enable more than just "freebsd-*/amd64" with this change, > are the other combinations tested / working ? > > Thanks > > -- > Vincent Legoll > -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Sat May 19 13:12:03 2018 From: armin.rigo at gmail.com (Armin Rigo) Date: Sat, 19 May 2018 19:12:03 +0200 Subject: [pypy-dev] [PATCH] FreeBSD build fix In-Reply-To: References: Message-ID: Hi David, On 18 May 2018 at 16:09, David CARLIER wrote: > Hi dear list, I have hard time to fork the project thus I m sendng this > little patch to fix FreeBSD build. As far as I can see, the function ``detect_cpu.autodetect()`` never returns the string 'amd64'. Are you sure this part of the patch is meaningful? A bient?t, Armin. From devnexen at gmail.com Sun May 20 12:39:18 2018 From: devnexen at gmail.com (David CARLIER) Date: Sun, 20 May 2018 17:39:18 +0100 Subject: [pypy-dev] [PATCH] FreeBSD build fix Message-ID: Here a new version of the patch. Kind regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: diff Type: application/octet-stream Size: 1181 bytes Desc: not available URL: From hubo at jiedaibao.com Mon May 21 22:46:21 2018 From: hubo at jiedaibao.com (hubo) Date: Tue, 22 May 2018 10:46:21 +0800 Subject: [pypy-dev] Mysterious IndexError in service running with PyPy In-Reply-To: <5A3C662E.9070506@jiedaibao.com> References: <5a37757a.4a98500a.ba397.d553SMTPIN_ADDED_BROKEN@mx.google.com> <5A3C662E.9070506@jiedaibao.com> Message-ID: <5B03847A.1030703@jiedaibao.com>+BC2DA5E7A99BDD6A Update: I think I found the cause, happily it is not a PyPy bug (although it is PyPy-related). Some generators are calling some cleanup code in the "finally" clause, but they are not correctly collected in PyPy until GC (because PyPy does not use reference counting). Apparently when they are collected by GC module, the "finally" clause is executed in a separated thread. 2018-05-22 hubo ????"hubo" ?????2017-12-22 09:55 ???Re: [pypy-dev] Mysterious IndexError in service running with PyPy ????"William ML Leslie" ???"PyPy Developer Mailing List" It is a regular guess, but this method is guareenteed to be called only in the main thread. Thread pool usage is very limited in this program because it is coroutine-based. 2017-12-22 hubo ????William ML Leslie ?????2017-12-19 20:14 ???Re: [pypy-dev] Mysterious IndexError in service running with PyPy ????"hubo" ???"PyPy Developer Mailing List" Where is the code that changes the size of self.heap? How do we know that size(self.heap) is constant? My guess is that some thread changes this; but l is not recomputed. On 18 Dec 2017 6:59 PM, "hubo" wrote: I'm reporting this issue in this mail group, though I don't know if it is related with PyPy, because it is really strange, and is not able to reproduce stably. But I hope someone may know the reason or have some points. I'm running some SDN services written in Python with PyPy 5.3.0. The related code is here: https://github.com/hubo1016/vlcp/blob/8022e3a3c67cf4305af503d507640a730ca3949d/vlcp/utils/indexedheap.py#L100 The full code is also in the repo, but may be too complex to describe. But this related piece is quite simple: def _siftdown(self, pos): temp = self.heap[pos] l = len(self.heap) while pos * 2 + 1 < l: cindex = pos * 2 + 1 pt = self.heap[cindex] if cindex + 1 < l and self.heap[cindex+1][0] < pt[0]: cindex = cindex + 1 pt = self.heap[cindex] if pt[0] < temp[0]: self.heap[pos] = pt self.index[pt[1]] = pos else: break pos = cindex self.heap[pos] = temp self.index[temp[1]] = pos It is a simple heap operation. The service uses a heap to process timers. When the service is not busy, it usually runs this piece of code several times per minute. I have 32 servers running this service. They are quite stable in about three months, but one day one of the services crashes on line 100 reporting IndexError: pt = self.heap[cindex] As you can see, cindex = pos * 2 + 1, which is tested by the while pre-conditon just two lines before. And there is not any multi-threading issues here because this piece of code always runs in the same thread. So it is not possible in theory for this to happen. Only the following facts are known about this issue: It reproduces - through quite rarely. I've met this kind of crashes 4 times, each with different machine, so it should not be related to hardware issuess. Since I've got 32 servers, it might take more than one year to reproduce with a single server. It seems not to be related to pressures. All of the crashes happens at night, when there are little requests. Only some cleanup tasks are running in fixed interval. The services are running with PyPy 5.3.0. I've upgraded a few of them to 5.9, but it will take a long time to validate whether this still happens. And It is not validated on CPython too. I'm also trying to collect more debugging information for this issue, but it is very hard since it rarely reproduces. It is not a serious issue. It could be workarounded with a auto-restart, but I'm searching the cause. 2017-12-18 hubo _______________________________________________ pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Tue May 22 02:21:57 2018 From: armin.rigo at gmail.com (Armin Rigo) Date: Tue, 22 May 2018 08:21:57 +0200 Subject: [pypy-dev] Mysterious IndexError in service running with PyPy In-Reply-To: <5b0384c5.1c69fb81.d8569.2e54SMTPIN_ADDED_BROKEN@mx.google.com> References: <5a37757a.4a98500a.ba397.d553SMTPIN_ADDED_BROKEN@mx.google.com> <5A3C662E.9070506@jiedaibao.com> <5b0384c5.1c69fb81.d8569.2e54SMTPIN_ADDED_BROKEN@mx.google.com> Message-ID: Hi, On 22 May 2018 at 04:46, hubo wrote: > I think I found the cause, happily it is not a PyPy bug (although it is > PyPy-related). Some generators are calling some cleanup code in the > "finally" clause, but they are not correctly collected in PyPy until GC > (because PyPy does not use reference counting). Apparently when they are > collected by GC module, the "finally" clause is executed in a separated > thread. Unlike Java, we're not purposefully running finalizers in its own threads: instead, if your program has got more than one thread running, finalizers will run in the one where the GC happens to run. There isn't much we can do about that. All these hard GC issues also exist on CPython, but they are just more hidden. Typically, if there is a cycle of objects which itself holds, indirectly, a reference to the generator-iterator, then you get the same effect: the cycle will be broken by the GC module of CPython at a random point in time and from a random thread, too. So we can argue that you potentially have a bug even on CPython, in the sense that an apparently unrelated change somewhere else can make the same bug appear on CPython. A bient?t, Armin. From hubo at jiedaibao.com Tue May 22 03:15:22 2018 From: hubo at jiedaibao.com (hubo) Date: Tue, 22 May 2018 15:15:22 +0800 Subject: [pypy-dev] Mysterious IndexError in service running with PyPy In-Reply-To: References: <5a37757a.4a98500a.ba397.d553SMTPIN_ADDED_BROKEN@mx.google.com> <5A3C662E.9070506@jiedaibao.com> <5b0384c5.1c69fb81.d8569.2e54SMTPIN_ADDED_BROKEN@mx.google.com> Message-ID: <5B03C387.4020506@jiedaibao.com>+E78C0CB65B51204C Thank you for your explaination! One more thing I would like to confirm: looks like in PyPy3, a "yield from my_generator()" expression delegates all exceptions inside, so the finalizer in my_generator() is always executed. Am I right? 2018-05-22 hubo ????Armin Rigo ?????2018-05-22 14:21 ???Re: [pypy-dev] Mysterious IndexError in service running with PyPy ????"hubo" ???"William ML Leslie","PyPy Developer Mailing List" Hi, On 22 May 2018 at 04:46, hubo wrote: > I think I found the cause, happily it is not a PyPy bug (although it is > PyPy-related). Some generators are calling some cleanup code in the > "finally" clause, but they are not correctly collected in PyPy until GC > (because PyPy does not use reference counting). Apparently when they are > collected by GC module, the "finally" clause is executed in a separated > thread. Unlike Java, we're not purposefully running finalizers in its own threads: instead, if your program has got more than one thread running, finalizers will run in the one where the GC happens to run. There isn't much we can do about that. All these hard GC issues also exist on CPython, but they are just more hidden. Typically, if there is a cycle of objects which itself holds, indirectly, a reference to the generator-iterator, then you get the same effect: the cycle will be broken by the GC module of CPython at a random point in time and from a random thread, too. So we can argue that you potentially have a bug even on CPython, in the sense that an apparently unrelated change somewhere else can make the same bug appear on CPython. A bient?t, Armin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From divyam.agg22 at gmail.com Thu May 31 08:35:15 2018 From: divyam.agg22 at gmail.com (divyam.agg22) Date: Thu, 31 May 2018 05:35:15 -0700 (MST) Subject: [pypy-dev] About gurobipy In-Reply-To: References: Message-ID: <1527770115282-0.post@n5.nabble.com> Hi, My name is Divyam and I am working on a large-scale discrete optimization problem. I have used Pypy earlier in order to improve the runtime of my python code. Now I am using some external packages in python such as Gurobipy, Docplex. Is it possible to use these packages with Pypy and if yes, how can I install these packages in Pypy? Thanks and Regards, Divyam -- Sent from: http://pypy.1116445.n5.nabble.com/ From armin.rigo at gmail.com Thu May 31 11:53:04 2018 From: armin.rigo at gmail.com (Armin Rigo) Date: Thu, 31 May 2018 17:53:04 +0200 Subject: [pypy-dev] About gurobipy In-Reply-To: <1527770115282-0.post@n5.nabble.com> References: <1527770115282-0.post@n5.nabble.com> Message-ID: Hi, On 31 May 2018 at 14:35, divyam.agg22 wrote: > My name is Divyam and I am working on a large-scale discrete optimization > problem. I have used Pypy earlier in order to improve the runtime of my > python code. Now I am using some external packages in python such as > Gurobipy, Docplex. Is it possible to use these packages with Pypy and if > yes, how can I install these packages in Pypy? Is the C/C++ source code of the Gurobipy extension module available anywhere? I didn't find it in 2015. If it is not, then it can't be used with PyPy because PyPy requires recompilation. A bient?t, Armin.