From michal at bendowski.pl Wed Aug 1 21:20:54 2012 From: michal at bendowski.pl (Michal Bendowski) Date: Wed, 1 Aug 2012 21:20:54 +0200 Subject: [pypy-dev] jvm-improvements branch Message-ID: Hello, At this point it's probably pretty clear I wasn't working on PyPy full time (you heard from me once every two months...) but nevertheless I would like to ask a few questions: 1. How can the buildbot be triggered? My work on the JVM backend is already in the jvm-improvements branch, it should now finally build and work against current default. If I remember correctly, Antonio has setup the appropriate 32bit chroot with JPype for the branch. 2. What is the general test-passing policy? I tried running the tests myself (to see if I have introduced any regressions), but lots of unrelated tests failed on my OS X box. Is this an unsupported OS problem, or are they suppose to fail on default for now? 3. If my code from jvm-improvements looks good I would love to get it merged into default, to close one chapter of work (a working rjvm module) and start working on whatever's next. Can someone take a look? Cheers, Micha? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjenvey at underboss.org Wed Aug 1 22:29:11 2012 From: pjenvey at underboss.org (Philip Jenvey) Date: Wed, 1 Aug 2012 13:29:11 -0700 Subject: [pypy-dev] jvm-improvements branch In-Reply-To: References: Message-ID: On Aug 1, 2012, at 12:20 PM, Michal Bendowski wrote: > Hello, > > At this point it's probably pretty clear I wasn't working on PyPy full time (you heard from me once every two months...) but nevertheless I would like to ask a few questions: > > 1. How can the buildbot be triggered? My work on the JVM backend is already in the jvm-improvements branch, it should now finally build and work against current default. If I remember correctly, Antonio has setup the appropriate 32bit chroot with JPype for the branch. > > 2. What is the general test-passing policy? I tried running the tests myself (to see if I have introduced any regressions), but lots of unrelated tests failed on my OS X box. Is this an unsupported OS problem, or are they suppose to fail on default for now? > You can force a build on this page (and a branch can be specified): http://buildbot.pypy.org/builders You'd ideally want all the tests passing before merging to default of course. A buildbot run should let you know if the failures on your platform are just noise, anyway. -- Philip Jenvey From aidembb at yahoo.com Tue Aug 7 07:41:05 2012 From: aidembb at yahoo.com (Roger Flores) Date: Mon, 6 Aug 2012 22:41:05 -0700 (PDT) Subject: [pypy-dev] Fatal RPython error: MemoryError Message-ID: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> I'm still getting MemoryError exceptions in RPython, as I mentioned a few months back.? I only knew of one way to reproduce it back then, but now I'm seeing different ways.? So, inspired, I wrote a short program to quickly exhaust memory in the hopes of reproducing the problem.? It seems to work! Fast! If you run the code pasted at the bottom with CPython, it should end printing 'test passed'. If you run the code with pypy 1.9 (I tried Windows), you should get: RPython traceback: ? File "translator_goal_targetpypystandalone.c", line 1048, in entry_point ? File "interpreter_function.c", line 929, in funccall__star_1 ? File "interpreter_function.c", line 958, in funccall__star_1 ? File "rpython_memory_gc_minimark.c", line 2518, in MiniMarkGC_collect_and_rese rve ? File "rpython_memory_gc_minimark.c", line 2220, in MiniMarkGC_minor_collection ? File "rpython_memory_gc_minimark.c", line 4258, in MiniMarkGC_collect_oldrefs_ to_nursery ? File "rpython_memory_gc_base.c", line 1799, in trace___trace_drag_out ? File "rpython_memory_gc_minimark.c", line 6667, in MiniMarkGC__malloc_out_of_n ursery_nonsmall Fatal RPython error: MemoryError Can someone else at least confirm this? Thanks, -Roger Flores exhaust-mem.py: """ Code to exhaust memory and generate a MemoryError exception """ size = 1 # find the largest memory allocation that succeeds try: ??? while True: ??????? new_size = size * 2 ??????? d = [0] * size ??????? size = new_size ??? except MemoryError as msg: ??? pass l = [] # exhaust memory while True: ??? try: ??????? # add the allocations to a list so references to them remain. ??????? # otherwise the garbage collector will clean them out! ??????? l.append([0] * size) ??????? print '{:,}'.format(size) ??? ??? except MemoryError as msg: ??????? if size > 1: ??????????? # can't allocate this much memory anymore.? Try a smaller size ??????????? size = size / 2 ??????? else: ??????????? print 'test passed' ??????????? break From arigo at tunes.org Tue Aug 7 09:15:21 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 7 Aug 2012 09:15:21 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> Message-ID: Hi Roger, On Tue, Aug 7, 2012 at 7:41 AM, Roger Flores wrote: > Can someone else at least confirm this? Yes, I can reproduce it easily running in a 32-bit chroot on a machine with more than 4GB of RAM. The problem is the same, and is still not solved: we run out of memory when doing a minor cycle of garbage collection because we don't have enough RAM to even copy the surviving objects outside the nursery. We can try to fix this particular problem, which requires looking in details through the GC for what occurs if any malloc fails during a collection. A bient?t, Armin. From arigo at tunes.org Tue Aug 7 10:09:43 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 7 Aug 2012 10:09:43 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> Message-ID: Re-hi, I tried to debug it more precisely, and it seems that the problem is even more basic than I thought. It is very hard to solve in general. The issue is that when we are really out of memory, then *every* single allocation is going to fail. The difference with CPython is that in the same situation, the latter can still (randomly) satisfy a small number of allocations of different sizes. This is an internal detail of how the memory allocators work. What occurs then in PyPy is that when we are out of memory, we can really not allocate any single object at more. So we cannot even execute anything from the "except MemoryError" block, because when catching the exception, we try to allocate a small internal object --- which re-raises MemoryError. I tried to look if we would go anywhere by making sure that no small internal object is allocated, but I doubt it, because then it would crash in the first line (e.g. "size = size / 2", which of course allocates a new "long" object). In the end you get this "fatal RPython error: MemoryError" because even printing the traceback requires some memory (and thus re-raises MemoryError instead). Generally, we have no hope to pass cleanly the test you gave. Even in CPython it works by chance; I can tweak it in "reasonable" ways and have it fail too (e.g. if you need to do a bit more than "size = size / 2" here, then you're likely to get a MemoryError in the except handler too). So, I don't think this is ever going to be fixed (or fixable)... A bient?t, Armin. From chef at ghum.de Tue Aug 7 10:23:57 2012 From: chef at ghum.de (Massa, Harald Armin) Date: Tue, 7 Aug 2012 10:23:57 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> Message-ID: > What occurs then in PyPy is that when we are out of memory, we can > really not allocate any single object at more. outside of programming there is the concept of having a secret backup ("nest egg"). Would'nt it be a solution to preallocate some bytes on startup for really, really bad times? As "really, really bad times" this kind of memory error would qualify. So there should be made a special case "if you are out of memory, to handle this out of memory error, use the preallocated bytes from the secret backup you made on startup". Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 From jacob at openend.se Tue Aug 7 10:40:42 2012 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Tue, 7 Aug 2012 10:40:42 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> Message-ID: <201208071040.43004.jacob@openend.se> Tuesday 07 August 2012 you wrote: > Re-hi, > > I tried to debug it more precisely, and it seems that the problem is > even more basic than I thought. It is very hard to solve in general. > The issue is that when we are really out of memory, then *every* > single allocation is going to fail. The difference with CPython is > that in the same situation, the latter can still (randomly) satisfy a > small number of allocations of different sizes. This is an internal > detail of how the memory allocators work. > > What occurs then in PyPy is that when we are out of memory, we can > really not allocate any single object at more. So we cannot even > execute anything from the "except MemoryError" block, because when > catching the exception, we try to allocate a small internal object --- > which re-raises MemoryError. I tried to look if we would go anywhere > by making sure that no small internal object is allocated, but I doubt > it, because then it would crash in the first line (e.g. "size = size / > 2", which of course allocates a new "long" object). In the end you > get this "fatal RPython error: MemoryError" because even printing the > traceback requires some memory (and thus re-raises MemoryError > instead). > > Generally, we have no hope to pass cleanly the test you gave. Even in > CPython it works by chance; I can tweak it in "reasonable" ways and > have it fail too (e.g. if you need to do a bit more than "size = size > / 2" here, then you're likely to get a MemoryError in the except > handler too). > > So, I don't think this is ever going to be fixed (or fixable)... Can't you have a chunk of reserve memory allocated from the start of the GC, which you free if a memory allocation fails? Jacob From arigo at tunes.org Tue Aug 7 11:28:14 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 7 Aug 2012 11:28:14 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> Message-ID: Hi Harald, On Tue, Aug 7, 2012 at 10:23 AM, Massa, Harald Armin wrote: > outside of programming there is the concept of having a secret backup > ("nest egg"). Would'nt it be a solution to preallocate some bytes on > startup for really, really bad times? It might help in general, but not in this case: this example causes and catches multiple MemoryErrors. If you unveil your secret backup at the first MemoryError, you gained nothing, because the crash occurs only at the ~25th MemoryError. A bient?t, Armin. From Ronny.Pfannschmidt at gmx.de Tue Aug 7 13:56:45 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 07 Aug 2012 13:56:45 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> Message-ID: <5021027D.4050504@gmx.de> Hi Armin, Harald, On 08/07/2012 11:28 AM, Armin Rigo wrote: > Hi Harald, > > On Tue, Aug 7, 2012 at 10:23 AM, Massa, Harald Armin wrote: >> outside of programming there is the concept of having a secret backup >> ("nest egg"). Would'nt it be a solution to preallocate some bytes on >> startup for really, really bad times? > > It might help in general, but not in this case: this example causes > and catches multiple MemoryErrors. If you unveil your secret backup > at the first MemoryError, you gained nothing, because the crash occurs > only at the ~25th MemoryError. > I'm under the Impression that that Kind of Error could be evaded by reserving a special part of the memory pool for strings and internal objects *after* the general Pool is used up. My assumption is that most of the time the Memory errors are caused semi-large allocations/reallocation (like allocate or re-size a list/array/matrix) in which case there will still be some memory left to at least print a stack-trace at interpreter level and quit. However in some cases (like the example) the memory error is caused by accumulating more and more smaller objects, in which case the memory one would use for a Stack-trace is used up and the extra pool would be used. In that case i presume any allocation not needed for a trace would cause just another memory error in the exception handler. That should be able to handle most of the normal cases of people trying to handle something and print out some extra information before failing Its of course not safe from malicious code or people just deliberately refusing to fail if there is a memory-error. but really "evil" code is fine to receive a rpython traceback. -- Ronny > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Tue Aug 7 15:33:02 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 7 Aug 2012 15:33:02 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: <5021027D.4050504@gmx.de> References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> <5021027D.4050504@gmx.de> Message-ID: Hi Ronny, On Tue, Aug 7, 2012 at 1:56 PM, Ronny Pfannschmidt wrote: > However in some cases (like the example) the memory error is caused by > accumulating more and more smaller objects, > in which case the memory one would use for a Stack-trace is used up and the > extra pool would be used. I'm quite unsure how you propose to do that. We could have a reserve pool that we activate when trying to print the fatal MemoryError, but that doesn't really help: the normal stack trace cannot be even *built* in memory, as it requires allocating PyTraceback objects. That's the reason why the RPython traceback reported at the start of this thread is so short: because all other levels in the stack caught the RPython MemoryError, but failed very quickly when trying to handle it, throwing *another* RPython MemoryError to the caller. If you can come up with a more precise scheme, you're welcome. The issue is to know when it's ok to reserve from that pool and when we should raise an RPython MemoryError instead. A possible answer would look like "when allocating internal vs user objects", for some unspecified-yet distinction of internal vs user objects. A bient?t, Armin. From wickedgrey at gmail.com Tue Aug 7 19:25:14 2012 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Tue, 7 Aug 2012 10:25:14 -0700 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> <5021027D.4050504@gmx.de> Message-ID: On Tue, Aug 7, 2012 at 6:33 AM, Armin Rigo wrote: > If you can come up with a more precise scheme, you're welcome. The > issue is to know when it's ok to reserve from that pool and when we > should raise an RPython MemoryError instead. A possible answer would > look like "when allocating internal vs user objects", for some > unspecified-yet distinction of internal vs user objects. Would having all allocations inside an except MemoryError: block be eligible to use the reserved pool be workable? Or perhaps "inside an except: block, while a MemoryError is the current exception"? It might even be ok to just have it be "while inside an except: block". To my naive eyes, it seems like something simple like one of the above would handle the normal use cases, and allow for some real-world use to provide information about if the criteria need tightening, etc. Cheers, Eli From aidembb at yahoo.com Wed Aug 8 00:23:57 2012 From: aidembb at yahoo.com (Roger Flores) Date: Tue, 7 Aug 2012 15:23:57 -0700 (PDT) Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> <5021027D.4050504@gmx.de> Message-ID: <1344378237.57525.YahooMailNeo@web45514.mail.sp1.yahoo.com> On Tue, Aug 7, 2012 at 6:33 AM, Armin Rigo wrote: > If you can come up with a more precise scheme, you're welcome.? The > issue is to know when it's ok to reserve from that pool and when we > should raise an RPython MemoryError instead.? A possible answer would I'm not convinced this MemoryError is unsolvable yet.? Perhaps it can be broken into a couple of steps. When the app requests memory that would leave too little remaining, the memory request needs to be refused and a MemoryError sent instead.? This informs the app that it can no longer make memory requests like that and expect them to succeed. The app then has a few options.? Leave the exception unhandled and get terminated.? Reduce it's memory usage by pruning it's structures. Stop an operation and report the problem, i.e. "Document too large too load".? The key point is that the RPython memory system signals the app's memory system so it can respond as needed. If the app tries to handle the MemoryError it will need more memory requests to succeed, for temporaries like Armin points out, to get it's job done.? Either the free memory available will be restored by the app to an amount sufficient to avoid further MemoryErrors, or the app basically had it's chance and now RPython needs the remaining memory to successfully describe the apps early demise (print MemoryError and a stack trace showing what the app was doing). So we don't need special pools of memory, new APIs or anything special.? We do need to have two out of memory thresholds.? The first is enough for RPython to terminate the app and describe the problem.? The second, higher threshold, is used to notify the app that it can no longer request memory like that unless it reduces it's usage.? Only one MemoryError is sent to the app until either the app frees enough memory or gets terminated. Imaginary code could look something like this: def alloc(size): ??? if free_mem - size < app_reserve + rpython_reserve: ??????? ??????? if not raised_MemoryError: ??????????? raised_MemoryError = True ??????????? raise MemoryError() ??????? ??????? if free_mem - size < rpython_reserve: ??????????? print 'MemoryError' ??????????? print stack_trace() # of app ??????????????? ??? else: ??????? raised_MemoryError = False??? ??????????? ??????? app_reserve and rpython_reserve might be 1K, 10K, 100K, idk.? Apps might want a variable they can tweak to further pad the reserve because they know they need extra room to prune their memory systems. My main question is can the RPython memory system do a collection while using only rpython_reserve of memory? Note the MemoryError when RPython terminates the app isn't really a MemoryError.? It's not an exception.? It can't be caught (it's too late for that).? A different name really should be used to avoid confusion. Basically, this design sends one MemoryError to my test code to give it a chance to do something about it.? And it will let it keep going for a while.? Soon though, it will give up on the test code,? terminate it, and report why.? No RPython traceback is needed.? To me, the key issue is that apps get at least one MemoryError so they can do something about it.? The current state, that they may or may not get to do *anything* is kind of scary. -Roger From arigo at tunes.org Wed Aug 8 10:09:44 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 8 Aug 2012 10:09:44 +0200 Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: <1344378237.57525.YahooMailNeo@web45514.mail.sp1.yahoo.com> References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> <5021027D.4050504@gmx.de> <1344378237.57525.YahooMailNeo@web45514.mail.sp1.yahoo.com> Message-ID: Hi Eli, hi Roger, Both of your solutions might work. Roger, you overlook a problem in this pseudo-code: > print 'MemoryError' > print stack_trace() # of app We can't suddenly print the stack trace of the application from inside the GC. In order to have a regular stack trace printed, an exception must be raised, which is caught and re-raised by the various levels, collecting stack information in (new) PyTraceback objects; then, in PyPy, we need to import (at app-level) the "traceback" module and call a function from there. This takes a lot of extra memory. Well, overall we need first someone to care about sanitizing the GC first. When the GC raises an RPython MemoryError in the middle of a minor collection, it probably leaves things in a broken state. It didn't show up so far because you can't do anything anyway but get the crash. A bient?t, Armin. From aidembb at yahoo.com Wed Aug 8 18:54:43 2012 From: aidembb at yahoo.com (Roger Flores) Date: Wed, 8 Aug 2012 09:54:43 -0700 (PDT) Subject: [pypy-dev] Fatal RPython error: MemoryError In-Reply-To: References: <1344318065.38936.YahooMailNeo@web45513.mail.sp1.yahoo.com> <5021027D.4050504@gmx.de> <1344378237.57525.YahooMailNeo@web45514.mail.sp1.yahoo.com> Message-ID: <1344444883.7796.YahooMailNeo@web45505.mail.sp1.yahoo.com> >Well, overall we need first someone to care about sanitizing the GC >first.? When the GC raises an RPython MemoryError in the middle of a >minor collection, it probably leaves things in a broken state.? It Again, perhaps this could be avoided for now.? Could the rpython_reserve be set high enough so that minor collections always succeed?? 10KB?? 100KB?? Basically get it working as desired first, then go back and optimize memory sizes and collection cycles when someone can. Fun stat: pypy for my PPM app can store twice as much data as python in the 2GB of RAM!? So using a silly huge number for the reserve memories still puts pypy well ahead in the memory issues.? That is, if python doesn't need to be more careful as well. >We can't suddenly print the stack trace of the application from inside ... >a function from there.? This takes a lot of extra memory. OK.? It sounds like it already works, just needs a large rpython_reserve.? Again, pick a large number, show it works, optimize later? -Roger ________________________________ From: Armin Rigo To: Roger Flores Cc: "pypy-dev at python.org" Sent: Wednesday, August 8, 2012 1:09 AM Subject: Re: [pypy-dev] Fatal RPython error: MemoryError Hi Eli, hi Roger, Both of your solutions might work. Roger, you overlook a problem in this pseudo-code: >? ? ? ? ? ? print 'MemoryError' >? ? ? ? ? ? print stack_trace() # of app We can't suddenly print the stack trace of the application from inside the GC.? In order to have a regular stack trace printed, an exception must be raised, which is caught and re-raised by the various levels, collecting stack information in (new) PyTraceback objects; then, in PyPy, we need to import (at app-level) the "traceback" module and call a function from there.? This takes a lot of extra memory. Well, overall we need first someone to care about sanitizing the GC first.? When the GC raises an RPython MemoryError in the middle of a minor collection, it probably leaves things in a broken state.? It didn't show up so far because you can't do anything anyway but get the crash. A bient?t, Armin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Aug 8 19:05:16 2012 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Aug 2012 18:05:16 +0100 Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature Request) In-Reply-To: References: Message-ID: On Tue, Jul 31, 2012 at 12:25 PM, Steven Jackson wrote: > I know there is no current plan to implement scipy in pypy, but searching > the PyPy website, I was not able to find the reason. > > If it is not to be included as a feature due to lack of interest or > developer time, I am offering to begin rewriting scipy in pure python (I > have quite a bit of free time to do this). Pretty much everything in scipy depends on some fortran libraries (scipy.linalg), so the first step would be to have some system that allow you to automatically wrap fortran through cffi. It is quite a pain because of fortran/C ABI issues, but you could focus on one platform only at first. David From matti.picus at gmail.com Wed Aug 8 19:32:27 2012 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 08 Aug 2012 20:32:27 +0300 Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature Request) In-Reply-To: References: Message-ID: <5022A2AB.1060808@gmail.com> Someone tried getting f2py working on pypy a while ago, a quick search turned up http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html and a fork of code at https://bitbucket.org/pypy/f2pypy Perhaps that could be a starting point? Matti On 8/08/2012 8:05 PM, David Cournapeau wrote: > On Tue, Jul 31, 2012 at 12:25 PM, Steven Jackson > wrote: >> I know there is no current plan to implement scipy in pypy, but searching >> the PyPy website, I was not able to find the reason. >> >> If it is not to be included as a feature due to lack of interest or >> developer time, I am offering to begin rewriting scipy in pure python (I >> have quite a bit of free time to do this). > Pretty much everything in scipy depends on some fortran libraries > (scipy.linalg), so the first step would be to have some system that > allow you to automatically wrap fortran through cffi. It is quite a > pain because of fortran/C ABI issues, but you could focus on one > platform only at first. > > David > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From tim.tadh at gmail.com Wed Aug 8 20:12:14 2012 From: tim.tadh at gmail.com (Tim Henderson) Date: Wed, 8 Aug 2012 14:12:14 -0400 Subject: [pypy-dev] Dependence Graphs in Pypy? Message-ID: Hi Pypy devs! I am a graduate student working on mining software repositories, specifically mining programming patterns (or rules) from code bases.[1] I would like to do some work mining patterns from Python programs and I could jump start that work if I could generate dependence graphs from python source files (or from the bytecode). So I was wondering if Pypy does this already. Actually it would be a big help even if pypy only generated a control flow graph. From my quick skim of some of the source code it seems like it does this for RPython but I wasn't sure about regular python. If I could get a CFG I could then annotate it with the data dependence edges. Any pointers to where I should look in the Pypy code would be appreciated! Additionally, if you feel having python pDG's would be helpful in Pypy's interpreter perhaps any work I do towards that end could be a starting point. Thanks for the excellent work on Pypy! [1] Here is some of my research group's previous work on pattern extraction http://doi.wiley.com/10.1002/smr.532 (if you can't get the paper and would like it let me know) Cheers --- Tim Henderson mail me: tim.tadh at gmail.com mail me: timothy.henderson at case.edu github: github.com/timtadh -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Wed Aug 8 20:14:36 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 8 Aug 2012 11:14:36 -0700 Subject: [pypy-dev] Dependence Graphs in Pypy? In-Reply-To: References: Message-ID: On Wed, Aug 8, 2012 at 11:12 AM, Tim Henderson wrote: > Hi Pypy devs! > > I am a graduate student working on mining software repositories, > specifically mining programming patterns (or rules) from code bases.[1] I > would like to do some work mining patterns from Python programs and I could > jump start that work if I could generate dependence graphs from python > source files (or from the bytecode). > > So I was wondering if Pypy does this already. Actually it would be a big > help even if pypy only generated a control flow graph. From my quick skim > of some of the source code it seems like it does this for RPython but I > wasn't sure about regular python. If I could get a CFG I could then > annotate it with the data dependence edges. > > Any pointers to where I should look in the Pypy code would be appreciated! > Additionally, if you feel having python pDG's would be helpful in Pypy's > interpreter perhaps any work I do towards that end could be a starting > point. > > Thanks for the excellent work on Pypy! > > [1] Here is some of my research group's previous work on pattern > extraction http://doi.wiley.com/10.1002/smr.532 (if you can't get the > paper and would like it let me know) > > Cheers > --- > Tim Henderson > mail me: tim.tadh at gmail.com > mail me: timothy.henderson at case.edu > github: github.com/timtadh > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > You're correct, we only generate full CFGs for RPython. The Python bytecode compiler for PyPy (and CPython) does have a CFG, but they're just local to functions so I don't think they're what you're looking for. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From aterrel at tacc.utexas.edu Wed Aug 8 20:52:14 2012 From: aterrel at tacc.utexas.edu (Andy Ray Terrel) Date: Wed, 8 Aug 2012 13:52:14 -0500 Subject: [pypy-dev] Invitation to participate in SIAM CS&E Minisymposium Message-ID: Hello all, I'm putting together a minisymposium at SIAM CS&E 2013 ( http://www.siam.org/meetings/cse13/ ) on Fast Numerical Code in High Level Languages. Having a speaker from your community would be a wonderful addition to our minisymposium. Unfortunately tight deadlines means I need an answer by Friday (Aug 10). Can any of you come? If you agree please provide a draft title of your presentation. We will then follow up with instructions for you to submit a 75-word abstract to the conference system. Unfortunately, SIAM does not provide travel support for minisymposium speakers. Title: Fast numerical computing with high-level languages Abstract: The scientific computing workflow includes many advanced tasks that require fast specialized tools. Researchers have responded through the use of high-level languages that hide many of the complexities in each of these tools. Unfortunately this leaves a dichotomy in the development of applications with high level code piping to the low-level tools with numerous practical problems. We present many successful approaches to mitigate this two level coding hierarchy. The approaches range from Just In Time (JIT) compiling, meta-programming tools, and better wrapping tools with domain specific knowledge. -- Andy -- Andy R. Terrel, Ph.D. Texas Advanced Computing Center University of Texas at Austin aterrel at tacc.utexas.edu From arigo at tunes.org Wed Aug 8 21:05:57 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 8 Aug 2012 21:05:57 +0200 Subject: [pypy-dev] Dependence Graphs in Pypy? In-Reply-To: References: Message-ID: Hi Tim, An additional remark: if you're looking for a tool that is able to extract a complete call graph from a random Python program, then it's impossible. Only approximations can be done, like e.g. done in pylint, I believe. Such tools are pointless in PyPy but could be useful in other projects. For example I can imagine a tool that would statically compile some version of a random Python program, including all necessary guards to check at run-time that the assumptions made are correct. When these guards fail, it would fall back to a regular interpreter. The difference with PyPy is that the latter uses a tracing JIT compiler to do at run-time (partially) the same job as I describe above. It does it by observation of the run-time behavior, which is a rather better estimator of the program's general behavior than a complex static analysis. And it dispenses us from writing any such analysis. This gives us a result that is not specific to a particular version of Python (or to Python at all). A bient?t, Armin. From leonard.de.haro at ens.fr Wed Aug 8 21:52:55 2012 From: leonard.de.haro at ens.fr (=?ISO-8859-1?Q?L=E9onard_de_Haro?=) Date: Wed, 08 Aug 2012 15:52:55 -0400 Subject: [pypy-dev] Help (v.concretetype is NOT lltype.Void) In-Reply-To: References: <4FE87842.3070209@ens.fr> Message-ID: <5022C397.5010705@ens.fr> Hi everyone, For those who don't remember, I'm an undergrad stutent doing an internship about writing textbook interpreters and turn them into JITing VMs using the pypy translation toolchaine. I'm facing a little problem I don't understand. I've written a version of my interpreter using Map as described in the paper Runtime feedback (http://dl.acm.org/citation.cfm?id=2069181 ) to replace dictionnaries I used to use, but when I try to translate I get "assert v.concretetype is lltype.Void" Assertion Error I know this error, I already faced it and going into Pypy's code, I found out this was about the order of variables in jitdriver.jit_merge_point and jitdriver.can_enter_jit . Previously changing approprietly the order of variables solved the problem, but this time even bruteforcely testing every permutation didn't solve the problem and I'm stuck. If you could help, I'll be very grateful. You can find the file here: https://github.com/zebign1/RPython-internship/blob/master/Interpreters/ifF1WAE/properOnes/withMap/JITRPinterpretIter.py Map implementation can be found in the file treeClass.py in the parent directory, all along with the parser file. Thanks for your help Leonard. From arigo at tunes.org Wed Aug 8 23:15:36 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 8 Aug 2012 23:15:36 +0200 Subject: [pypy-dev] Help (v.concretetype is NOT lltype.Void) In-Reply-To: <5022C397.5010705@ens.fr> References: <4FE87842.3070209@ens.fr> <5022C397.5010705@ens.fr> Message-ID: Hi L?onard, On Wed, Aug 8, 2012 at 9:52 PM, L?onard de Haro wrote: > "assert v.concretetype is lltype.Void" > Assertion Error After a bit of debugging, I found out that this is caused by the two variable "funMap" and "env" which are both known statically to be the EMPTY_MAP_ENV instance. It's a bug in PyPy that it causes such an obscure crash. It's also very likely to be a bug in your program, though. You would have found it immediately if you had tests (or less immediately if the translation finished and the program gave nonsense). If you really don't want to write tests, you should at least try to run the program directly on some reasonable examples before you try to translate it. A bient?t, Armin. From jonathan at slenders.be Wed Aug 8 23:27:01 2012 From: jonathan at slenders.be (Jonathan Slenders) Date: Wed, 8 Aug 2012 23:27:01 +0200 Subject: [pypy-dev] _marshal EOF error? Message-ID: Hi everyone, I have almost a sandlib.py library ready for the Twisted Matrix framework. (The sandlib which is shipped with PyPy is not usable within Twisted.) However, the deserialisation of marshalled messages fails when the sandbox Python process does very big write system calls. Actually, the problem is that the write calls arrive in multiple chunks in the outReceived of the ProcessProtocol (Twisted code). Marshal.loads thinks the first chunk is valid on its own while it isn't yet. Long story short, there is a difference between the Python built-in and the pure python _marshall.py from PyPy which should be used for the sandbox hypervisor. It should throw EOF when a string-object is truncated, while it doesn't. >>> import _marshal # lib_pypy/_marshal.py >>> _marshal.loads('(\x02\x00\x00\x00i\x03\x00\x00\x00sB\xf9\x00\x00\n>> >>> import marshal # Python built-in >>> marshal.loads('(\x02\x00\x00\x00i\x03\x00\x00\x00sB\xf9\x00\x00\n", line 1, in EOFError: EOF read where object expected The patch, just add this one line to lib_pypy/_marshal.py (line 433) def _read(self, n): pos = self.bufpos newpos = pos + n * if newpos > len(self.bufstr): raise EOFError* ret = self.bufstr[pos : newpos] self.bufpos = newpos return ret This is the first bug that I found. Should I create a bug report or patch file for this? Thanks, Jonathan Slenders -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Aug 8 23:59:00 2012 From: cournape at gmail.com (David Cournapeau) Date: Wed, 8 Aug 2012 22:59:00 +0100 Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature Request) In-Reply-To: <5022A2AB.1060808@gmail.com> References: <5022A2AB.1060808@gmail.com> Message-ID: On Wed, Aug 8, 2012 at 6:32 PM, Matti Picus wrote: > Someone tried getting f2py working on pypy a while ago, a quick search > turned up > http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html > and a fork of code at > https://bitbucket.org/pypy/f2pypy > > Perhaps that could be a starting point? Yes, definitely, although there are tools which are better architected like fwrap (https://github.com/kwmsmith/fwrap). I would expect something based on f2py to be the fastest route, though. fwrap would more likely lend to something that could be contributed by and useful to both cpython and pypy-based implementations. David From arigo at tunes.org Thu Aug 9 00:02:48 2012 From: arigo at tunes.org (Armin Rigo) Date: Thu, 9 Aug 2012 00:02:48 +0200 Subject: [pypy-dev] _marshal EOF error? In-Reply-To: References: Message-ID: Hi Jonathan, On Wed, Aug 8, 2012 at 11:27 PM, Jonathan Slenders wrote: > The patch, just add this one line to lib_pypy/_marshal.py (line 433) Thanks! Checked in this extra line together with a 2-line test in lib_pypy/pypy_test/test_marshal_extra.py. For larger changes or ones that are not so "obviously correct", you would need to submit either a patch to the bug tracker https://bugs.pypy.org/ , or a pull request on bitbucket if you are familiar with "hg". A bient?t, Armin. From tim.tadh at gmail.com Thu Aug 9 02:28:34 2012 From: tim.tadh at gmail.com (Tim Henderson) Date: Wed, 8 Aug 2012 20:28:34 -0400 Subject: [pypy-dev] Dependence Graphs in Pypy? In-Reply-To: References: Message-ID: On Wed, Aug 8, 2012 at 3:05 PM, Armin Rigo wrote: > Hi Tim, > > An additional remark: if you're looking for a tool that is able to > extract a complete call graph from a random Python program, then it's > impossible. Only approximations can be done, like e.g. done in > pylint, I believe. > > Such tools are pointless in PyPy but could be useful in other > projects. For example I can imagine a tool that would statically > compile some version of a random Python program, including all > necessary guards to check at run-time that the assumptions made are > correct. When these guards fail, it would fall back to a regular > interpreter. > > The difference with PyPy is that the latter uses a tracing JIT > compiler to do at run-time (partially) the same job as I describe > above. It does it by observation of the run-time behavior, which is a > rather better estimator of the program's general behavior than a > complex static analysis. And it dispenses us from writing any such > analysis. This gives us a result that is not specific to a particular > version of Python (or to Python at all). > > > A bient?t, > > Armin. > Hi Armin, Thanks for the lengthy reply. Fortunately, I don't need a complete call graph. To get started I really only need procedure level dependency graphs (with control and data-dependencies). This graph is essentially a CFG with extra edges for different types of data dependencies. Thanks to Alex's previous reply I am taking a look at the CPython bytecode compiler to see if I can extract the local CFG's it produces. I could see a static approach potentially beating the dynamic approach prior to the jit getting *hot* but after that it seems like the dynamic approach would win in the long run. Luckily, I don't have to worry about such things as I have no intention of writing a compiler (well for python anyway you guys seem to have that under control), my main interest lies in studying the evolution of the structure of the programs themselves. Thanks for the pointers! -Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From william.leslie.ttg at gmail.com Thu Aug 9 06:29:03 2012 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Thu, 9 Aug 2012 14:29:03 +1000 Subject: [pypy-dev] Dependence Graphs in Pypy? In-Reply-To: References: Message-ID: On 09/08/2012, Tim Henderson wrote: > Fortunately, I don't need a complete call graph. To get started I really > only need procedure level dependency graphs (with control and > data-dependencies). This graph is essentially a CFG with extra edges for > different types of data dependencies. Thanks to Alex's previous reply I am > taking a look at the CPython bytecode compiler to see if I can extract the > local CFG's it produces. Although pypy's own compiler creates a block structure and allows you to do post-order traversal of the blocks, I've found the easiest way to get the CFG of a single code object to be to use the logic found within the dis module. Here's an example of some old code that does that, somewhat stripped down and provided without tests I'm afraid because I kind of got bored of it. It's slightly more involved than dis.findlabels, part of a large piece of nonsense for deriving dataflow equations from python at app-level. https://gist.github.com/3300866 Functions OP_FN are not hard to specify but I've got them intermingled with other, much more complicated code here, they use the EffectContext to store the residual behaviour. Stack depth is completely statically determinable, and in 2.5 or later, it's determinable without looking at other code objects. I think other people have managed to do this, by the way - I have the vague recollection that Prof. Yanhong Liu at Stony Brook wrote a tool to extract datalog facts from python, although with hand-coded flow for built-ins; if you're only interested in intraprocedural analysis (often next-to-useless in python) I guess that would be fine. But I can't seem to find that, now. -- William Leslie From stefan_ml at behnel.de Thu Aug 9 07:50:50 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Aug 2012 07:50:50 +0200 Subject: [pypy-dev] Questions Regarding Scipy in Pypy (Not a Feature Request) In-Reply-To: References: <5022A2AB.1060808@gmail.com> Message-ID: David Cournapeau, 08.08.2012 23:59: > On Wed, Aug 8, 2012 at 6:32 PM, Matti Picus wrote: >> Someone tried getting f2py working on pypy a while ago, a quick search >> turned up >> http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html >> and a fork of code at >> https://bitbucket.org/pypy/f2pypy >> >> Perhaps that could be a starting point? > > Yes, definitely, although there are tools which are better architected > like fwrap (https://github.com/kwmsmith/fwrap). > > I would expect something based on f2py to be the fastest route, > though. fwrap would more likely lend to something that could be > contributed by and useful to both cpython and pypy-based > implementations. FWIW, another reason why fwrap is popular, besides a better design, is that it can automatically generate Cython declarations for a Fortran wrapper, so it already supports two targets. Adding PyPy as a third one would reduce the need for users to learn new tools. Stefan From jonathan at slenders.be Thu Aug 9 09:52:44 2012 From: jonathan at slenders.be (Jonathan Slenders) Date: Thu, 9 Aug 2012 09:52:44 +0200 Subject: [pypy-dev] _marshal EOF error? In-Reply-To: References: Message-ID: Merci Armin! Keep up the great work! 2012/8/9 Armin Rigo > Hi Jonathan, > > On Wed, Aug 8, 2012 at 11:27 PM, Jonathan Slenders > wrote: > > The patch, just add this one line to lib_pypy/_marshal.py (line 433) > > Thanks! Checked in this extra line together with a 2-line test in > lib_pypy/pypy_test/test_marshal_extra.py. > > For larger changes or ones that are not so "obviously correct", you > would need to submit either a patch to the bug tracker > https://bugs.pypy.org/ , or a pull request on bitbucket if you are > familiar with "hg". > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Aug 9 16:03:56 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 9 Aug 2012 16:03:56 +0200 Subject: [pypy-dev] Invitation to participate in SIAM CS&E Minisymposium In-Reply-To: References: Message-ID: On Wed, Aug 8, 2012 at 8:52 PM, Andy Ray Terrel wrote: > Hello all, > > I'm putting together a minisymposium at SIAM CS&E 2013 ( > http://www.siam.org/meetings/cse13/ ) on Fast Numerical Code in High > Level Languages. Having a speaker from your community would be a > wonderful addition to our minisymposium. Unfortunately tight deadlines > means I need an answer by Friday (Aug 10). Can any of you come? > > If you agree please provide a draft title of your presentation. We > will then follow up with instructions for you to submit a 75-word > abstract to the conference system. Unfortunately, SIAM does not > provide travel support for minisymposium speakers. > > > Title: Fast numerical computing with high-level languages > > Abstract: > > The scientific computing workflow includes many advanced tasks that require > fast specialized tools. Researchers have responded through the use of > high-level languages that hide many of the complexities in each of these > tools. Unfortunately this leaves a dichotomy in the development of applications > with high level code piping to the low-level tools with numerous > practical problems. > > We present many successful approaches to mitigate this two level coding > hierarchy. The approaches range from Just In Time (JIT) compiling, > meta-programming tools, and better wrapping tools with domain specific > knowledge. > > > -- Andy > > -- > Andy R. Terrel, Ph.D. > Texas Advanced Computing Center > University of Texas at Austin > aterrel at tacc.utexas.edu > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev Hi I would be potentially inclined to come. Is there any assistance for travel and accomodation available for speakers? Cheers, fijal From aterrel at tacc.utexas.edu Thu Aug 9 16:58:49 2012 From: aterrel at tacc.utexas.edu (Andy Ray Terrel) Date: Thu, 9 Aug 2012 09:58:49 -0500 Subject: [pypy-dev] Invitation to participate in SIAM CS&E Minisymposium In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 9:03 AM, Maciej Fijalkowski wrote: > On Wed, Aug 8, 2012 at 8:52 PM, Andy Ray Terrel wrote: >> Hello all, >> >> I'm putting together a minisymposium at SIAM CS&E 2013 ( >> http://www.siam.org/meetings/cse13/ ) on Fast Numerical Code in High >> Level Languages. Having a speaker from your community would be a >> wonderful addition to our minisymposium. Unfortunately tight deadlines >> means I need an answer by Friday (Aug 10). Can any of you come? >> >> If you agree please provide a draft title of your presentation. We >> will then follow up with instructions for you to submit a 75-word >> abstract to the conference system. Unfortunately, SIAM does not >> provide travel support for minisymposium speakers. >> >> >> Title: Fast numerical computing with high-level languages >> >> Abstract: >> >> The scientific computing workflow includes many advanced tasks that require >> fast specialized tools. Researchers have responded through the use of >> high-level languages that hide many of the complexities in each of these >> tools. Unfortunately this leaves a dichotomy in the development of applications >> with high level code piping to the low-level tools with numerous >> practical problems. >> >> We present many successful approaches to mitigate this two level coding >> hierarchy. The approaches range from Just In Time (JIT) compiling, >> meta-programming tools, and better wrapping tools with domain specific >> knowledge. >> >> >> -- Andy >> >> -- >> Andy R. Terrel, Ph.D. >> Texas Advanced Computing Center >> University of Texas at Austin >> aterrel at tacc.utexas.edu >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > Hi > > I would be potentially inclined to come. Is there any assistance for > travel and accomodation available for speakers? > > Cheers, > fijal Hi Fijal, That would be great if you could come. Unfortunately the only support available is for students ( http://www.siam.org/prizes/sponsored/travel.php ). -- Andy -- Andy R. Terrel, Ph.D. Texas Advanced Computing Center University of Texas at Austin aterrel at tacc.utexas.edu From fijall at gmail.com Thu Aug 9 17:23:14 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 9 Aug 2012 17:23:14 +0200 Subject: [pypy-dev] Invitation to participate in SIAM CS&E Minisymposium In-Reply-To: References: Message-ID: On Thu, Aug 9, 2012 at 4:58 PM, Andy Ray Terrel wrote: > On Thu, Aug 9, 2012 at 9:03 AM, Maciej Fijalkowski wrote: >> On Wed, Aug 8, 2012 at 8:52 PM, Andy Ray Terrel wrote: >>> Hello all, >>> >>> I'm putting together a minisymposium at SIAM CS&E 2013 ( >>> http://www.siam.org/meetings/cse13/ ) on Fast Numerical Code in High >>> Level Languages. Having a speaker from your community would be a >>> wonderful addition to our minisymposium. Unfortunately tight deadlines >>> means I need an answer by Friday (Aug 10). Can any of you come? >>> >>> If you agree please provide a draft title of your presentation. We >>> will then follow up with instructions for you to submit a 75-word >>> abstract to the conference system. Unfortunately, SIAM does not >>> provide travel support for minisymposium speakers. >>> >>> >>> Title: Fast numerical computing with high-level languages >>> >>> Abstract: >>> >>> The scientific computing workflow includes many advanced tasks that require >>> fast specialized tools. Researchers have responded through the use of >>> high-level languages that hide many of the complexities in each of these >>> tools. Unfortunately this leaves a dichotomy in the development of applications >>> with high level code piping to the low-level tools with numerous >>> practical problems. >>> >>> We present many successful approaches to mitigate this two level coding >>> hierarchy. The approaches range from Just In Time (JIT) compiling, >>> meta-programming tools, and better wrapping tools with domain specific >>> knowledge. >>> >>> >>> -- Andy >>> >>> -- >>> Andy R. Terrel, Ph.D. >>> Texas Advanced Computing Center >>> University of Texas at Austin >>> aterrel at tacc.utexas.edu >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >> >> Hi >> >> I would be potentially inclined to come. Is there any assistance for >> travel and accomodation available for speakers? >> >> Cheers, >> fijal > > Hi Fijal, > > That would be great if you could come. Unfortunately the only support > available is for students ( > http://www.siam.org/prizes/sponsored/travel.php ). > > -- Andy > > > -- > Andy R. Terrel, Ph.D. > Texas Advanced Computing Center > University of Texas at Austin > aterrel at tacc.utexas.edu Hi I would like to come, but I'll absolutely not make commitments right now, 9 months in advance. Noone will pay me to do that and I'm incapable of saying if flying halfway across the world from South Africa in March is feasible or not. Cheers, fijal From arigo at tunes.org Fri Aug 10 10:38:21 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 10 Aug 2012 10:38:21 +0200 Subject: [pypy-dev] Invitation to participate in SIAM CS&E Minisymposium In-Reply-To: References: Message-ID: Hi Fijal, On Thu, Aug 9, 2012 at 5:23 PM, Maciej Fijalkowski wrote: > I would like to come, but I'll absolutely not make commitments right > now, 9 months in advance. Noone will pay me to do that and I'm > incapable of saying if flying halfway across the world from South > Africa in March is feasible or not. Fwiw, Pycon US is also in March, 11 days after. Armin From tbaldridge at gmail.com Fri Aug 10 23:06:45 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Fri, 10 Aug 2012 16:06:45 -0500 Subject: [pypy-dev] custom interpreter split_block error Message-ID: I have a interpreter that translates fine, but when I turn on the JIT I get the following error [translation:ERROR] File "/home/tim/pypy/pypy/jit/metainterp/warmspot.py", line 266, in split_graph_and_record_jitdriver [translation:ERROR] graph.startblock = support.split_before_jit_merge_point(*jmpp) [translation:ERROR] File "/home/tim/pypy/pypy/jit/codewriter/support.py", line 79, in split_before_jit_merge_point [translation:ERROR] link = split_block(None, portalblock, 0, greens_v + reds_v) [translation:ERROR] File "/home/tim/pypy/pypy/translator/unsimplify.py", line 105, in split_block [translation:ERROR] assert v.concretetype is lltype.Void [translation:ERROR] AssertionError My merge point looks like this: jitdriver.jit_merge_point(ip = self._ip, func = self._call_stack[-1], args = self._arg_stack[-1], stack = self._sp) >From what I can tell of the source, this is something to do with one of my variables infact being a constant and not a variable. What am I doing wrong? I'm generating my bytecode at runtime via an AST: def run_benchmark(times): max = Argument("max") cur = Argument("cur") ast = If(Equal(max, cur), cur, Call(CurFunc(), Add(cur, Const(W_Int(1))), max)) f = Func([cur, max], ast) #value = Interpreter(f.toFunction()).main_loop(W_Int(0), W_Int(2147483648)) value = Interpreter(f.toFunction()).main_loop(W_Int(0), W_Int(int(times))) return value Could it be that the translator is propagating my function through as a constant? I.e. I'm basically hard-coding my byte code via the AST. Thank for any help, Timothy Baldridge -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri Aug 10 23:15:05 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 10 Aug 2012 23:15:05 +0200 Subject: [pypy-dev] custom interpreter split_block error In-Reply-To: References: Message-ID: Hi, On Fri, Aug 10, 2012 at 11:06 PM, Timothy Baldridge wrote: > jitdriver.jit_merge_point(ip = self._ip, > func = self._call_stack[-1], > args = self._arg_stack[-1], > stack = self._sp) > > > From what I can tell of the source, this is something to do with one of my > variables infact being a constant and not a variable. What am I doing wrong? It's impossible to answer you without seeing the source of the whole interpreter. You have to know precisely which of the four variables turned into a constant; then it should be easy to follow back why. Constantness can only follow from translation-time constants, but sometimes a bit indirectly (e.g. maybe self._arg_stack is a list in which you only ever put one translation-time constant). A bient?t, Armin. From fijall at gmail.com Sat Aug 11 09:26:53 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 11 Aug 2012 09:26:53 +0200 Subject: [pypy-dev] custom interpreter split_block error In-Reply-To: References: Message-ID: On Fri, Aug 10, 2012 at 11:15 PM, Armin Rigo wrote: > Hi, > > On Fri, Aug 10, 2012 at 11:06 PM, Timothy Baldridge > wrote: >> jitdriver.jit_merge_point(ip = self._ip, >> func = self._call_stack[-1], >> args = self._arg_stack[-1], >> stack = self._sp) >> >> >> From what I can tell of the source, this is something to do with one of my >> variables infact being a constant and not a variable. What am I doing wrong? > > It's impossible to answer you without seeing the source of the whole > interpreter. You have to know precisely which of the four variables > turned into a constant; then it should be easy to follow back why. > Constantness can only follow from translation-time constants, but > sometimes a bit indirectly (e.g. maybe self._arg_stack is a list in > which you only ever put one translation-time constant). > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev You get a similar message when you just forget to list an alive variable in jit_merge_point. From fijall at gmail.com Sat Aug 11 10:43:39 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 11 Aug 2012 10:43:39 +0200 Subject: [pypy-dev] jvm tests failing on os x Message-ID: Anyone has any idea how to fix/disable them? An example: http://buildbot.pypy.org/summary/longrepr?testname=TestJvmException.%28%29.test_multiple_prebuilt_dead_weakrefs&builder=own-macosx-x86-32&build=984&mod=translator.jvm.test.test_weakref Cheers, fijal From arigo at tunes.org Sat Aug 11 11:28:16 2012 From: arigo at tunes.org (Armin Rigo) Date: Sat, 11 Aug 2012 11:28:16 +0200 Subject: [pypy-dev] jvm tests failing on os x In-Reply-To: References: Message-ID: Hi Fijal, On Sat, Aug 11, 2012 at 10:43 AM, Maciej Fijalkowski wrote: > Anyone has any idea how to fix/disable them? Obviously you can disable them by saying "if sys.platform = 'darwin': py.test.skip('XXX')" a few lines before the crash, but is that really your question? Armin From fijall at gmail.com Sat Aug 11 12:11:47 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 11 Aug 2012 12:11:47 +0200 Subject: [pypy-dev] jvm tests failing on os x In-Reply-To: References: Message-ID: On Sat, Aug 11, 2012 at 11:28 AM, Armin Rigo wrote: > Hi Fijal, > > On Sat, Aug 11, 2012 at 10:43 AM, Maciej Fijalkowski wrote: >> Anyone has any idea how to fix/disable them? > > Obviously you can disable them by saying "if sys.platform = 'darwin': > py.test.skip('XXX')" a few lines before the crash, but is that really > your question? > > > Armin I was more thinking about fixing but fine :) From pjenvey at underboss.org Sun Aug 12 02:29:34 2012 From: pjenvey at underboss.org (Philip Jenvey) Date: Sat, 11 Aug 2012 17:29:34 -0700 Subject: [pypy-dev] jvm tests failing on os x In-Reply-To: References: Message-ID: <3C66E291-9370-481D-8CBE-C6F681F939A1@underboss.org> On Aug 11, 2012, at 1:43 AM, Maciej Fijalkowski wrote: > Anyone has any idea how to fix/disable them? > > An example: > > http://buildbot.pypy.org/summary/longrepr?testname=TestJvmException.%28%29.test_multiple_prebuilt_dead_weakrefs&builder=own-macosx-x86-32&build=984&mod=translator.jvm.test.test_weakref The cause is we're using a Java 6 Math.copySign method and the buildbot must be Java 5. (as shown here: http://buildbot.pypy.org/builders/own-macosx-x86-32/builds/984/steps/shell_6/logs/stdio ) That wouldn't be too difficult to workaround but we could also consider requiring Java 6 (or even 7) as a minimum. Java 5 is EOL'd and e.g. Jython 2.7 requires Java 6. Michael, do you have any Java 6 stuff in the jvm-improvements branch? -- Philip Jenvey From arigo at tunes.org Sun Aug 12 10:37:28 2012 From: arigo at tunes.org (Armin Rigo) Date: Sun, 12 Aug 2012 10:37:28 +0200 Subject: [pypy-dev] jvm-improvements branch In-Reply-To: References: Message-ID: Hi Michal, On Wed, Aug 1, 2012 at 9:20 PM, Michal Bendowski wrote: > 3. If my code from jvm-improvements looks good I would love to get it merged > into default, to close one chapter of work (a working rjvm module) and start > working on whatever's next. Can someone take a look? I did a review of your work based only on "does it look like it breaks anything outside of JVM/CLI". I have only one comment: the ctypes_configure package should not contain any "import pypy.xxx". Apart from that, it looks like it wouldn't break anything so I'm fine if this is merged. A bient?t, Armin. From michal at bendowski.pl Mon Aug 13 13:44:39 2012 From: michal at bendowski.pl (Michal Bendowski) Date: Mon, 13 Aug 2012 13:44:39 +0200 Subject: [pypy-dev] jvm tests failing on os x In-Reply-To: <3C66E291-9370-481D-8CBE-C6F681F939A1@underboss.org> References: <3C66E291-9370-481D-8CBE-C6F681F939A1@underboss.org> Message-ID: On Sun, Aug 12, 2012 at 2:29 AM, Philip Jenvey wrote: > > > On Aug 11, 2012, at 1:43 AM, Maciej Fijalkowski wrote: > > > Anyone has any idea how to fix/disable them? > > > > An example: > > > > http://buildbot.pypy.org/summary/longrepr?testname=TestJvmException.%28%29.test_multiple_prebuilt_dead_weakrefs&builder=own-macosx-x86-32&build=984&mod=translator.jvm.test.test_weakref > > The cause is we're using a Java 6 Math.copySign method and the buildbot must be Java 5. > > (as shown here: http://buildbot.pypy.org/builders/own-macosx-x86-32/builds/984/steps/shell_6/logs/stdio ) > > That wouldn't be too difficult to workaround but we could also consider requiring Java 6 (or even 7) as a minimum. Java 5 is EOL'd and e.g. Jython 2.7 requires Java 6. > > Michael, do you have any Java 6 stuff in the jvm-improvements branch? I don't think I introduced any Java 6 specific code. Nevertheless Java 6 will be EOL'd soon as well (Scala is dropping Java 6 support in the next release for example) so I agree that requiring at least Java 6 seems like a good idea. Micha? From uwe.ng at family-hoffmann.de Mon Aug 13 15:49:25 2012 From: uwe.ng at family-hoffmann.de (Uwe Hoffmann) Date: Mon, 13 Aug 2012 15:49:25 +0200 Subject: [pypy-dev] cppyy comments/questions Message-ID: Hi, i'm not sure if this is the correct mailing list as this is more a question of "developing with" than "developing of" pypy. I made some tests with cppyy and it looks promising but I have some questions/comments. preface: Tested with trunk and the reflex-support branch. building with: python translate.py -O jit --gcrootfinder=shadowstack \ targetpypystandalone.py --withmod-cppyy 64-bit linux , gcc 4.6.1 1) During build/translation there were several errors like "undefined reference to `Reflex::Member::~Member()". This was caused by not linking against libReflex. With some modifications (some insertions of (lib,"Reflex") e.g. ) in pypy/translator/tool/cbuild.py i got a working (but see later) module. But what's the correct way of doing the build. The "cppyy use case question" thread mentions the problem but i didn't see the solution. 2) On importing some wrapped "c++ libs" the displayed error is very generic. The following patch gave me some hints what's wrong . diff -r 26b81a6d9365 pypy/module/cppyy/interp_cppyy.py --- a/pypy/module/cppyy/interp_cppyy.py Fri Jun 29 12:26:46 2012 +0200 +++ b/pypy/module/cppyy/interp_cppyy.py Fri Aug 10 23:30:30 2012 +0200 @@ -22,7 +22,7 @@ try: cdll = capi.c_load_dictionary(name) except rdynload.DLOpenError, e: - raise OperationError(space.w_RuntimeError, space.wrap(str(e))) + raise OperationError(space.w_RuntimeError,space.wrap(str(e.msg))) 3) playing with operators.h in pypy/module/cppyy/test/: (m_int made public before test) the following lines results with the expected output "10" import cppyy cppyy.load_reflection_info('operatorsDict.so') number = cppyy.gbl.number n=number(10) print n.m_int but if i add "virtual" to "operator==" like virtual bool operator==(const number& n) const {return m_int == n.m_int; } the output is not deterministic anymore. This needs to be confirmed as my setup is probably not correct (see point 1). 4) During one test i got the following error: TypeError: none of the 1 overloaded methods succeeded. Full details : A* B::C::methodName() => AttributeError: '' has no attribute '' (A and C are types (classes), B is a namespace) Generally speaking: What's the best way for debugging this problem. Regards, Uwe From tbaldridge at gmail.com Mon Aug 13 17:08:55 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 13 Aug 2012 10:08:55 -0500 Subject: [pypy-dev] Need some pointers on my interpreter Message-ID: I'm in the process of writing a interpreter in pypy and I'd like some pointers on how to optimize my inner loop for better jit optimization. My interpreter code is here: https://github.com/halgari/cljvm/blob/master/system/interpreter.py And the wrapper/unwrapper bits are here: https://github.com/halgari/cljvm/blob/master/system/objspace.py My test program is here: https://github.com/halgari/cljvm/blob/master/tests/benchmark/basic_math.py In this example the interpreter is getting "max" from the command line, then counting from 0 to "max" via using a tail call. This looks something like the following in Python: def f(cur, max): if cur == max: return cur else: f(cur + 1, max) Since this interpreter supports tail calls, the above code should basically be jitted down to a simple for loop, or even better, just returning "max". Here is my jitlog: https://gist.github.com/3341474 A few notes about the VM: Almost everything is immutable. That is, there are no local variables. Locals are created via creating an anonymous function and passing variables as arguments. There are several "stacks" in the VM: _call_stack - normal stack of functions _arg_stack - the arguments of the functions in _call_stack _stack - the main data stack. Temp variables are stored here _ip_stack - the current instruction pointer location of each function in _call_stack >From what I can tell from the jitlog, it looks like there's a ton of code being generated by the TAIL_CALL opcode. It seems the JIT doesn't realize that the int values going into a function are the same ones coming out of the previous iteration of the loop. In other words, there's no reason to box the arguments to the TAIL_CALL into W_Ints, or to put them into an array, since the next iteration of the loop will just take them out again. Any suggestions of what I could do better here? Thank you so much for your help, I've been playing with PyPy for years, and it's exciting to finally understand this code enough that I can at least make a change and see my benchmarks improve slightly. PyPy is an awesome project. Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Tue Aug 14 00:23:38 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Mon, 13 Aug 2012 15:23:38 -0700 (PDT) Subject: [pypy-dev] cppyy comments/questions In-Reply-To: References: Message-ID: Hi Uwe, > 1) During build/translation there were several errors > like "undefined reference to `Reflex::Member::~Member()". > This was caused by not linking against libReflex. > With some modifications (some insertions of > (lib,"Reflex") e.g. ) in pypy/translator/tool/cbuild.py > i got a working (but see later) module. But what's the correct > way of doing the build. The "cppyy use case question" thread > mentions the problem but i didn't see the solution. the original command that you had works for me, and I've not yet see any missing symbols problem. W/o being able to reproduce, it's hard to say anything sensible. There's probably something differently setup. Now that CFFI is in PyPy default, though, I hope to be able to completely get rid of the need to link pypy-c with libReflex (the library shouldn't be needed until "import cppyy" is done). > 2) On importing some wrapped "c++ libs" the displayed error is very > generic. The following patch gave me some hints what's wrong . > > diff -r 26b81a6d9365 pypy/module/cppyy/interp_cppyy.py > --- a/pypy/module/cppyy/interp_cppyy.py Fri Jun 29 12:26:46 2012 +0200 > +++ b/pypy/module/cppyy/interp_cppyy.py Fri Aug 10 23:30:30 2012 +0200 > @@ -22,7 +22,7 @@ > try: > cdll = capi.c_load_dictionary(name) > except rdynload.DLOpenError, e: > - raise OperationError(space.w_RuntimeError, space.wrap(str(e))) > + raise OperationError(space.w_RuntimeError,space.wrap(str(e.msg))) Thanks! I'll add this, together with a test to test_fragile.py, shortly. Most of the time, I use the automatic class loader these days, though. Saves on typing and fewer errors. > 3) playing with operators.h in pypy/module/cppyy/test/: > (m_int made public before test) > the following lines results with the expected output "10" > import cppyy > cppyy.load_reflection_info('operatorsDict.so') > number = cppyy.gbl.number > n=number(10) > print n.m_int > > but if i add "virtual" to "operator==" like > virtual bool operator==(const number& n) const {return m_int == n.m_int; > } It's not clear to me why this is so, but the problem shows up if some method is virtual while the destructor is not. Add a virtual dtor, and the problem goes away. Thanks, I'll look into this. It's easy to test. > 4) During one test i got the following error: > TypeError: none of the 1 overloaded methods succeeded. Full details : > A* B::C::methodName() => > AttributeError: '' has no attribute '' I'd like to have such a failing example to add to test_fragile.py and get a better diagnostic, but for this: > (A and C are types (classes), B is a namespace) > Generally speaking: What's the best way for debugging this problem. Most likely "A" isn't known or can not be build for some other reason. The __dir__ method on namespaces is implemented to show the string names of all known classes. If what I think is the case, then this: >>>> dir(cppyy.gbl) should show 'A', but this: >>>> cppyy.gbl.A will fail. Another possibility is that the code is really looking for the global namespace (which has name '', i.e. empty string, on the global namespace '::'), and that'd be a bug. But again, I'd really like the example so that a better diagnostic can be given. Thanks, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From brandon at rhodesmill.org Tue Aug 14 06:31:45 2012 From: brandon at rhodesmill.org (Brandon Rhodes) Date: Tue, 14 Aug 2012 00:31:45 -0400 Subject: [pypy-dev] what does the debug cycle look like? Message-ID: <87628mqeku.fsf@asaph.rhodesmill.org> I need some help. I want to help add more numpypy goodness to PyPy, so that I can rewrite my PyEphem astronomy module as pure Python instead of having to always maintain a C extension. I will then tell the users who are doing large calculations, and need extra speed, that they can run their script on PyPy and have it run blazing fast. My problem is that I am not sure how to start adding code to PyPy in such a way that I can try out my code more than once within a single work day, because it is taking 5+ hours to build the translator on every platform to which I have access. This did not surprise me on my 2003 Dell PowerEdge 400SC, with its 2GB of kind-of-working RAM, which wound up taking 5h 28m to complete the build. But I just got finished trying the build on the fastest computer in the house - my wife's few-month-old iMac with 4GB of memory - and it actually took *longer*, finishing up a full 5h 53m (!) after it started. And, actually, that's not the big problem. The big problem is that whereas C Python uses Make, so it only re-compiles the files I have changed when I am in a tight development loop, it seems that PyPy starts the whole translation process over again when I run it. How does one actually iterate when working on part of PyPy and its libraries? Only now that it is past midnight, as I write this, do I realize that my subsequent builds of PyPy can be done with the PyPy interpreter itself, speeding things up considerably! But even if I can bring down the build time to under a half-hour, I can hardly imagine doing software development under the constraint of such delays - what am I missing? My first feat, should it success, will be writing load() for numpypy since that it the first missing feature I noticed when I tried running my prototype "jplephem" package - my first attempt at doing heavy-duty astronomy in pure Python. I had hoped to have the package run and walk away with a number to compare to the pure C performance, but instead I get the fun of contributing! :) -- Brandon Rhodes brandon at rhodesmill.org http://rhodesmill.org/brandon From alex.gaynor at gmail.com Tue Aug 14 06:44:58 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 13 Aug 2012 21:44:58 -0700 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: <87628mqeku.fsf@asaph.rhodesmill.org> References: <87628mqeku.fsf@asaph.rhodesmill.org> Message-ID: On Mon, Aug 13, 2012 at 9:31 PM, Brandon Rhodes wrote: > I need some help. > > I want to help add more numpypy goodness to PyPy, so that I can rewrite > my PyEphem astronomy module as pure Python instead of having to always > maintain a C extension. I will then tell the users who are doing large > calculations, and need extra speed, that they can run their script on > PyPy and have it run blazing fast. > > My problem is that I am not sure how to start adding code to PyPy in > such a way that I can try out my code more than once within a single > work day, because it is taking 5+ hours to build the translator on every > platform to which I have access. This did not surprise me on my 2003 > Dell PowerEdge 400SC, with its 2GB of kind-of-working RAM, which wound > up taking 5h 28m to complete the build. But I just got finished trying > the build on the fastest computer in the house - my wife's few-month-old > iMac with 4GB of memory - and it actually took *longer*, finishing up a > full 5h 53m (!) after it started. > > And, actually, that's not the big problem. The big problem is that > whereas C Python uses Make, so it only re-compiles the files I have > changed when I am in a tight development loop, it seems that PyPy starts > the whole translation process over again when I run it. How does one > actually iterate when working on part of PyPy and its libraries? > > Only now that it is past midnight, as I write this, do I realize that my > subsequent builds of PyPy can be done with the PyPy interpreter itself, > speeding things up considerably! But even if I can bring down the build > time to under a half-hour, I can hardly imagine doing software > development under the constraint of such delays - what am I missing? > > My first feat, should it success, will be writing load() for numpypy > since that it the first missing feature I noticed when I tried running > my prototype "jplephem" package - my first attempt at doing heavy-duty > astronomy in pure Python. I had hoped to have the package run and walk > away with a number to compare to the pure C performance, but instead I > get the fun of contributing! :) > > -- > Brandon Rhodes brandon at rhodesmill.org > http://rhodesmill.org/brandon > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > The trick is tests! Tests can be run without translating (actually the whole interpreter can, but we like tests :D). Just run `pytest.py pypy/module/numpypy/` and you're running the tests, they take like a minute or two, which is way too long IMO, but it's totally serviceable for iterative development. The only reason we ever translate is to asses the performance of something. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Aug 14 07:09:31 2012 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 14 Aug 2012 08:09:31 +0300 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: <87628mqeku.fsf@asaph.rhodesmill.org> References: <87628mqeku.fsf@asaph.rhodesmill.org> Message-ID: <5029DD8B.50609@gmail.com> Welcome, your help is needed and appreciated. One thing you might want to try is adding the functionality as a pure python. You should be able to add a load function somewhere in the the lib_pypy/numpypy/core/directory, and add tests to the lib_pypy/numpypy/tests directory. Then just run (from the top pypy directory) pypy/translator/goal/pypy pytest.py lib_pypy/numpypy/tests Only if speed become your real bottleneck, or if you simply cannot implement your function (load in this case) in pure python, would you want to dive into the murky depths of rpython. Matti On 14/08/2012 7:31 AM, Brandon Rhodes wrote: > I need some help. > > I want to help add more numpypy goodness to PyPy, so that I can rewrite > my PyEphem astronomy module as pure Python instead of having to always > maintain a C extension. I will then tell the users who are doing large > calculations, and need extra speed, that they can run their script on > PyPy and have it run blazing fast. > > My problem is that I am not sure how to start adding code to PyPy in > such a way that I can try out my code more than once within a single > work day, because it is taking 5+ hours to build the translator on every > platform to which I have access. This did not surprise me on my 2003 > Dell PowerEdge 400SC, with its 2GB of kind-of-working RAM, which wound > up taking 5h 28m to complete the build. But I just got finished trying > the build on the fastest computer in the house - my wife's few-month-old > iMac with 4GB of memory - and it actually took *longer*, finishing up a > full 5h 53m (!) after it started. > > And, actually, that's not the big problem. The big problem is that > whereas C Python uses Make, so it only re-compiles the files I have > changed when I am in a tight development loop, it seems that PyPy starts > the whole translation process over again when I run it. How does one > actually iterate when working on part of PyPy and its libraries? > > Only now that it is past midnight, as I write this, do I realize that my > subsequent builds of PyPy can be done with the PyPy interpreter itself, > speeding things up considerably! But even if I can bring down the build > time to under a half-hour, I can hardly imagine doing software > development under the constraint of such delays - what am I missing? > > My first feat, should it success, will be writing load() for numpypy > since that it the first missing feature I noticed when I tried running > my prototype "jplephem" package - my first attempt at doing heavy-duty > astronomy in pure Python. I had hoped to have the package run and walk > away with a number to compare to the pure C performance, but instead I > get the fun of contributing! :) > From brandon at rhodesmill.org Tue Aug 14 07:11:22 2012 From: brandon at rhodesmill.org (Brandon Rhodes) Date: Tue, 14 Aug 2012 01:11:22 -0400 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: (Alex Gaynor's message of "Mon, 13 Aug 2012 21:44:58 -0700") References: <87628mqeku.fsf@asaph.rhodesmill.org> Message-ID: <871ujaqcqt.fsf@asaph.rhodesmill.org> Alex Gaynor writes: > The trick is tests! Tests can be run without translating (actually > the whole interpreter can, but we like tests :D). Just run `pytest.py > pypy/module/numpypy/` and you're running the tests, they take like a > minute or two, which is way too long IMO, but it's totally serviceable > for iterative development. The only reason we ever translate is to > asses the performance of something. Ah! That very nearly makes sense - tell me if this is what you meant: the tests are not run against the translated, baked-in version of numpypy sitting inside my pypy interpreter and that takes hours to rebuild, but against the raw numpypy code - both rpython and normal Python - that is sitting on the disk and that I can have just edited? -- Brandon Rhodes brandon at rhodesmill.org http://rhodesmill.org/brandon From brandon at rhodesmill.org Tue Aug 14 07:15:55 2012 From: brandon at rhodesmill.org (Brandon Rhodes) Date: Tue, 14 Aug 2012 01:15:55 -0400 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: <5029DD8B.50609@gmail.com> (Matti Picus's message of "Tue, 14 Aug 2012 08:09:31 +0300") References: <87628mqeku.fsf@asaph.rhodesmill.org> <5029DD8B.50609@gmail.com> Message-ID: <87wr12oxys.fsf@asaph.rhodesmill.org> Matti Picus writes: > Welcome, your help is needed and appreciated. One thing you might > want to try is adding the functionality as a pure python... Ah, yes, Maciej had mentioned something about that - that many NumPy functions wind up being pure Python even over in NumPy, and that for PyPy we can do the same thing. I will have to ask him - or do you know? - whether we are writing our own tests, or just copying over NumPy's existing tests each time we add a new feature and want to be sure that it works? -- Brandon Rhodes brandon at rhodesmill.org http://rhodesmill.org/brandon From arigo at tunes.org Tue Aug 14 10:34:32 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 14 Aug 2012 10:34:32 +0200 Subject: [pypy-dev] jvm tests failing on os x In-Reply-To: References: <3C66E291-9370-481D-8CBE-C6F681F939A1@underboss.org> Message-ID: Hi, On Mon, Aug 13, 2012 at 1:44 PM, Michal Bendowski wrote: > I don't think I introduced any Java 6 specific code. Nevertheless Java > 6 will be EOL'd soon as well (Scala is dropping Java 6 support in the > next release for example) so I agree that requiring at least Java 6 > seems like a good idea. So, can anyone that knows how add a few lines of code in translator/jvm, say in genjvm:detect_missing_support_programs(), to check not only that java and javac exist but are of a recent enough version? A bient?t, Armin. From fijall at gmail.com Tue Aug 14 10:59:43 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Aug 2012 10:59:43 +0200 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: <871ujaqcqt.fsf@asaph.rhodesmill.org> References: <87628mqeku.fsf@asaph.rhodesmill.org> <871ujaqcqt.fsf@asaph.rhodesmill.org> Message-ID: On Tue, Aug 14, 2012 at 7:11 AM, Brandon Rhodes wrote: > Alex Gaynor writes: > >> The trick is tests! Tests can be run without translating (actually >> the whole interpreter can, but we like tests :D). Just run `pytest.py >> pypy/module/numpypy/` and you're running the tests, they take like a >> minute or two, which is way too long IMO, but it's totally serviceable >> for iterative development. The only reason we ever translate is to >> asses the performance of something. > > Ah! That very nearly makes sense - tell me if this is what you meant: > the tests are not run against the translated, baked-in version of > numpypy sitting inside my pypy interpreter and that takes hours to > rebuild, but against the raw numpypy code - both rpython and normal > Python - that is sitting on the disk and that I can have just edited? > > -- > Brandon Rhodes brandon at rhodesmill.org http://rhodesmill.org/brandon > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev Yes. I appreciate you think we're super humans and can write code that can only be tested in 30min (not sure why your translation take so long), but we're not. We're very much into having a very short feedback loop. Cheers, fijal From arigo at tunes.org Tue Aug 14 11:09:51 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 14 Aug 2012 11:09:51 +0200 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: References: <87628mqeku.fsf@asaph.rhodesmill.org> <871ujaqcqt.fsf@asaph.rhodesmill.org> Message-ID: Hi, A quick note: On Tue, Aug 14, 2012 at 10:59 AM, Maciej Fijalkowski wrote: > (not sure why your translation take so long) Because of swapping: translating on a 64-bit Mac requires 5-6 GB of RAM ideally. A bient?t, Armin. From tbaldridge at gmail.com Tue Aug 14 13:38:42 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 14 Aug 2012 06:38:42 -0500 Subject: [pypy-dev] Need some pointers on my interpreter In-Reply-To: References: Message-ID: Here is my jitlog: > > https://gist.github.com/3341474 > > > I updated my jitlog after the work I did last night. I was able to pull about 50+ ops out of the jitted code, but still, I'm dealing with the fact that the JIT doesn't realize it can optimize away my data stack. Around line 55 in the stack, you can see that my IS_EQ op is still trying to update the stack, and the instruction pointer. So I guess this is my current question. What is the best way to represent a data stack (like python does) in a interpreter. Currently I have an array indexed by a stack pointer, and then I mark stack[sp] and stack[sp-1] as 'reds', but is that really what I should be doing here? Thanks, Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Aug 14 13:45:29 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Aug 2012 13:45:29 +0200 Subject: [pypy-dev] Need some pointers on my interpreter In-Reply-To: References: Message-ID: On Tue, Aug 14, 2012 at 1:38 PM, Timothy Baldridge wrote: > Here is my jitlog: >> >> >> https://gist.github.com/3341474 >> >> > I updated my jitlog after the work I did last night. I was able to pull > about 50+ ops out of the jitted code, but still, I'm dealing with the fact > that the JIT doesn't realize it can optimize away my data stack. Around line > 55 in the stack, you can see that my IS_EQ op is still trying to update the > stack, and the instruction pointer. > > So I guess this is my current question. What is the best way to represent a > data stack (like python does) in a interpreter. Currently I have an array > indexed by a stack pointer, and then I mark stack[sp] and stack[sp-1] as > 'reds', but is that really what I should be doing here? > > Thanks, > > Timothy > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > >From a quick glimpse it looks like you might be forced to use virtualziables here. this is just a very quick glimpse though From taavi.burns at gmail.com Tue Aug 14 14:25:08 2012 From: taavi.burns at gmail.com (Taavi Burns) Date: Tue, 14 Aug 2012 08:25:08 -0400 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: References: <87628mqeku.fsf@asaph.rhodesmill.org> <871ujaqcqt.fsf@asaph.rhodesmill.org> Message-ID: Also, that 4GB Mac looks like it has at least 4 cores, so Make will fork 4 cc processes, each of which wants 700-1000MB RAM. Especially if it's only 2-core with hyperthreading, you'll want to limit the concurrent compiles to 2 (I forget the exact args to `translate` at the moment, on the subway). And keep in mind that translating with PyPy takes more ram than cpython! -- taa /*eof*/ On 2012-08-14, at 5:09, Armin Rigo wrote: > Hi, > > A quick note: > > On Tue, Aug 14, 2012 at 10:59 AM, Maciej Fijalkowski wrote: >> (not sure why your translation take so long) > > Because of swapping: translating on a 64-bit Mac requires 5-6 GB of RAM ideally. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From tbaldridge at gmail.com Tue Aug 14 16:26:54 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 14 Aug 2012 09:26:54 -0500 Subject: [pypy-dev] Translator error on Windows: error C2036: 'void *' : unknown size Message-ID: I get the following error during translation on Windows: error C2036: 'void *' : unknown size I'm using 32 bit python to do my translation. Is there a known fix for this error? Thanks, Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From uwe.ng at family-hoffmann.de Tue Aug 14 16:40:19 2012 From: uwe.ng at family-hoffmann.de (Uwe Hoffmann) Date: Tue, 14 Aug 2012 16:40:19 +0200 Subject: [pypy-dev] cppyy comments/questions In-Reply-To: References: Message-ID: Am 14.08.2012 00:23, schrieb wlavrijsen at lbl.gov: > Hi Uwe, > >> 1) During build/translation there were several errors >> like "undefined reference to `Reflex::Member::~Member()". >> This was caused ". >> With some modifications (some insertions of >> (lib,"Reflex") e.g. ) in pypy/translator/tool/cbuild.py >> i got a working (but see later) module. But what's the correct >> way of doing the build. The "cppyy use case question" thread >> mentions the problem but i didn't see the solution. > > the original command that you had works for me, and I've not yet see any > missing symbols problem. W/o being able to reproduce, it's hard to say > anything sensible. There's probably something differently setup. > Hi Wim, well my first analysis was wrong when i said "by not linking against libReflex". the command line of the Makefile in /tmp/usession-reflex-support-*/ ( $(TARGET): $(OBJECTS)) actually places LDFLAGSEXTRA (containing -lReflex) in front of the object files. According to http://gcc.gnu.org/onlinedocs/gcc/Link-Options.html libReflex will not be used for searching missing symbols from *following* object files. I'm not sure if this is a g++/linker version dependent behaviour. rules in translator/platform/posix.py places $(LDFLAGSEXTRA) before $(OBJECTS) ('$(TARGET)', '$(OBJECTS)', '$(CC_LINK) $(LDFLAGSEXTRA) -o $@ $(OBJECTS) $(LIBDIRS) $(LIBS) $(LINKFILES) $(LDFLAGS)'), regards Uwe From arigo at tunes.org Tue Aug 14 17:31:58 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 14 Aug 2012 17:31:58 +0200 Subject: [pypy-dev] Translator error on Windows: error C2036: 'void *' : unknown size In-Reply-To: References: Message-ID: Hi Timothy, On Tue, Aug 14, 2012 at 4:26 PM, Timothy Baldridge wrote: > error C2036: 'void *' : unknown size Fixed yesterday evening in 9e9b39337354. Indeed, http://buildbot.pypy.org/nightly/trunk/ shows that Win32 has successfully built this night. A bient?t, Armin. From wlavrijsen at lbl.gov Tue Aug 14 18:51:14 2012 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Tue, 14 Aug 2012 09:51:14 -0700 (PDT) Subject: [pypy-dev] cppyy comments/questions In-Reply-To: References: Message-ID: Hi Uwe, > rules in translator/platform/posix.py places $(LDFLAGSEXTRA) before $(OBJECTS) > ('$(TARGET)', '$(OBJECTS)', '$(CC_LINK) $(LDFLAGSEXTRA) -o $@ $(OBJECTS) > $(LIBDIRS) $(LIBS) $(LINKFILES) $(LDFLAGS)'), ah, but that then just means that I should not use the 'link_extra' keyword in the ExternalCompilationInfo object (see cppyy/capi/reflex_capi.py) for -lReflex, but instead it should be part of $(LIBS), i.e. keyword 'libraries'. Will check that in shortly, after running a full translation. But again, the ECI is going to go away soon. Thanks, Wim --- a/pypy/module/cppyy/capi/reflex_capi.py Mon Aug 13 17:18:12 2012 -0700 +++ b/pypy/module/cppyy/capi/reflex_capi.py Tue Aug 14 09:48:58 2012 -0700 @@ -35,7 +35,7 @@ include_dirs=[incpath] + rootincpath, includes=["reflexcwrapper.h"], library_dirs=rootlibpath, - link_extra=["-lReflex"], + libraries=["Reflex"], use_cpp_linker=True, ) -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From matti.picus at gmail.com Tue Aug 14 20:29:05 2012 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 14 Aug 2012 21:29:05 +0300 Subject: [pypy-dev] what does the debug cycle look like? In-Reply-To: <87wr12oxys.fsf@asaph.rhodesmill.org> References: <87628mqeku.fsf@asaph.rhodesmill.org> <5029DD8B.50609@gmail.com> <87wr12oxys.fsf@asaph.rhodesmill.org> Message-ID: <502A98F1.4090601@gmail.com> Numpy's tests are a good starting point, but they tend to ignore corner cases such as: improper arguments, strange mixtures of options, passing in virtual ndarrays (lazy evaluation and slices), empty arrays or scalars, so we generally start with theirs and add more, trying to retain compatibility. Matti On 14/08/2012 8:15 AM, Brandon Rhodes wrote: > ... I will have to ask him - or do you know? - whether we are writing > our own tests, or just copying over NumPy's existing tests each time > we add a new feature and want to be sure that it works? From tbaldridge at gmail.com Wed Aug 15 05:45:41 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 14 Aug 2012 22:45:41 -0500 Subject: [pypy-dev] Need some pointers on my interpreter In-Reply-To: References: Message-ID: > > > From a quick glimpse it looks like you might be forced to use > virtualziables here. this is just a very quick glimpse though > So I've been playing around with this for about a day now, and I guess I don't understand the "vritualizable" requirements. When I try to mark a something as virtualizable in the JitDriver I get an error saying either: AttributeError: no field '__first_struct' or, I get an error about my virtualalizable type not being a Container. What are the requirements for these virtualizables? Thanks, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed Aug 15 17:38:18 2012 From: arigo at tunes.org (Armin Rigo) Date: Wed, 15 Aug 2012 17:38:18 +0200 Subject: [pypy-dev] Need some pointers on my interpreter In-Reply-To: References: Message-ID: Hi Timothy, On Wed, Aug 15, 2012 at 5:45 AM, Timothy Baldridge wrote: > So I've been playing around with this for about a day now, and I guess I > don't understand the "vritualizable" requirements. I also tried (although for 15 minutes only), but I gave up because the style is really too different from what we use in the Python interpreter. You have only one instance of Interpreter, and only one list that stores local variables for all functions. It's not really possible to have it work well with PyPy's JIT as far as it is now: it cannot at the same time "virtualize" the places near the top of the list (because that's where variables are currently accessed) and keep the rest of the list concrete. Instead, what we do in PyPy's Python interpreter is to have one Frame class per function, which is the obvious place to store a list of *this function*'s local variables. This would be what you need to mark "virtualizable". Closer to the style you are using is our regexp engine (pypy.rlib.rsre.rsre_core). It is based on only one "Interpreter"-like instance (in this case StrMatchContext) and has recursive invocations of the "interpreter main loop" using the same MatchContext instance. Looking for similarities, it declares as "greens" some attributes of this MatchContext (greens = [..., "ctx.pattern"]), which is a more efficient way than doing greens=["ip"] followed by "ip = self._ip" in the jit_merge_point. But it doesn't have any notion of storing all local variables of all frames of execution. At most it stores the regexp's marks, which is also a stack, but creating this stack is not optimized. I'm unsure if that model can work at all in your case without major efforts in the jit front-end. A bient?t, Armin. From andrew.elliott at spc.ox.ac.uk Wed Aug 15 21:22:10 2012 From: andrew.elliott at spc.ox.ac.uk (Andrew Elliott) Date: Wed, 15 Aug 2012 19:22:10 +0000 Subject: [pypy-dev] Issues with Networkx subgraph function - Slower than cPython Message-ID: <131E4EEC-BD8B-4C30-81EC-D26C4EFBE643@dtc.ox.ac.uk> Hello I have recently tried to move some code across to pypy from cPython, and it takes around 1.5x longer than the cPython implementation (~12 seconds against 19 seconds). Most of the time (90%) is spent in the multiple calls to the subgraph function of networkx. Is there anything that I can do to make pypy faster? I am running the Mac OSX 64 bit binary inside a virtualenv. Thanks, Andrew Elliott From fijall at gmail.com Wed Aug 15 21:44:42 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 15 Aug 2012 21:44:42 +0200 Subject: [pypy-dev] Issues with Networkx subgraph function - Slower than cPython In-Reply-To: <131E4EEC-BD8B-4C30-81EC-D26C4EFBE643@dtc.ox.ac.uk> References: <131E4EEC-BD8B-4C30-81EC-D26C4EFBE643@dtc.ox.ac.uk> Message-ID: On Wed, Aug 15, 2012 at 9:22 PM, Andrew Elliott wrote: > Hello > > I have recently tried to move some code across to pypy from cPython, and it takes around 1.5x longer than the cPython implementation (~12 seconds against 19 seconds). Most of the time (90%) is spent in the multiple calls to the subgraph function of networkx. Is there anything that I can do to make pypy faster? I am running the Mac OSX 64 bit binary inside a virtualenv. > > Thanks, > > Andrew Elliott You can provide us with more information. The best is a standalone example that we can run as small as possible. The next best is a way to set it up and reproduce. Without that we're seriously clueless. Cheers, fijal From Matt.Welch at compuware.com Fri Aug 17 15:04:07 2012 From: Matt.Welch at compuware.com (Welch, Matt) Date: Fri, 17 Aug 2012 13:04:07 +0000 Subject: [pypy-dev] Hello World in Windows with MinGW Message-ID: <2CF0DBBEB4B88F4EBA2CF49732D993C3117C2F46@BY2PRD0511MB441.namprd05.prod.outlook.com> Sorry to bother you with a newbie question. I'm trying to send a simple Hello World program through translate. It's failing to compile platcheck_1.c. I'm getting: [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c: In function `dump_section_PROCESS_QUERY_LIMITED_INFORMATION': [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:383: error: `PROCESS_QUERY_LIMITED_INFORMATION' undeclared (first use in this function) [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:383: error: (Each undeclared identifier is reported only once [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:383: error: for each function it appears in.) [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c: In function `dump_section_PROCESS_SUSPEND_RESUME': [platform:Error] c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\platcheck_1.c:403: error: `PROCESS_SUSPEND_RESUME' undeclared (first use in this function) I tried to figure out where the values in platcheck_1.c were getting defined in the first place so I could determine what I was doing wrong, but couldn't quite get a handle on it. I'm on Windows, running MinGW through Msys. I launched translate like this: pypy-1.9/pypy.exe pypy-pypy-341e1e3821ff/pypy/translator/goal/translate.py --cc=gcc --batch hello.py The compile line that is failing looks like this: [platform:execute] gcc -c -O3 c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\gcctest.c -o c:\docume~1\pfhmmw0\locals~1\temp\usession-release-1.9-15\gcctest.o Any help would be appreciated. Do I need a different environment? Is it not MinGW/Msys friendly? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wiktor8010 at o2.pl Sat Aug 18 19:41:23 2012 From: wiktor8010 at o2.pl (=?UTF-8?Q?Wiktor_Mizdal?=) Date: Sat, 18 Aug 2012 19:41:23 +0200 Subject: [pypy-dev] =?utf-8?q?ARM_backend_release?= Message-ID: <515e56a4.1a277920.502fd3c3.47eb4@o2.pl> When we expected a stable release of ARM backend? It possible in this year? Cheers, Wiktor ? ? From fijall at gmail.com Sun Aug 19 20:58:21 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 19 Aug 2012 20:58:21 +0200 Subject: [pypy-dev] ARM backend release In-Reply-To: <515e56a4.1a277920.502fd3c3.47eb4@o2.pl> References: <515e56a4.1a277920.502fd3c3.47eb4@o2.pl> Message-ID: On Sat, Aug 18, 2012 at 7:41 PM, Wiktor Mizdal wrote: > When we expected a stable release of ARM backend? > > It possible in this year? > > Cheers, > Wiktor ARM buildbot is green, which means it runs and works. Expect the next pypy release to officially support ARM. Cheers, fijal From pjenvey at underboss.org Thu Aug 23 02:32:44 2012 From: pjenvey at underboss.org (Philip Jenvey) Date: Wed, 22 Aug 2012 17:32:44 -0700 Subject: [pypy-dev] Developer selection for Py3k In-Reply-To: References: <4FECE3EC.7030000@gmail.com> Message-ID: On Jun 29, 2012, at 12:39 PM, Philip Jenvey wrote: > > On Jun 28, 2012, at 4:08 PM, Antonio Cuni wrote: > >> Hi all, >> >> as you probably know, the Py3k [1] proposal is getting funded thanks to our >> generous donors. >> >> During the first round, three of use were selected: me, Alex Gaynor and >> Benjamin Peterson. However, due to unforeseen unavailability of Alex and >> Benjamin, we are now looking for one more developer to help with the py3k work. >> >> If you are interested in getting paid to work on the Py3k proposal, please >> apply by replying to this email. >> >> To be applicable you need to be an experienced PyPy developer, preferably with >> a previous experience on the Python Interpreter part of PyPy. > > If it was a few months ago I'd be up for this, unfortunately I'm not available now. I'll let you know if that changes. I now have some free cycles available for Py3k work =] -- Philip Jenvey From stefan_ml at behnel.de Thu Aug 23 11:11:51 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 11:11:51 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Maciej Fijalkowski, 05.07.2012 14:50: > On Thu, Jul 5, 2012 at 2:35 PM, Stefan Behnel wrote: >> Maciej Fijalkowski, 05.07.2012 11:01: >>> On Thu, Jul 5, 2012 at 10:26 AM, Amaury Forgeot d'Arc wrote: >>>> 2012/7/5 Stefan Behnel >>>>> Back to that question then: >>>>> >>>>>> Is there a way to get readable debugging symbols in a translated PyPy >>>>>> that would tell me what is being executed? >>>> >>>> I fear that pypy standard distribution calls "strip" on the resulting >>>> binary. >>>> You could translate pypy yourself, I'm quite sure it contains debug info >>>> already and it's quite easy to call "make debug" anyway. >>> >>> Default build (not the distribution or nightly, you have to trasnlate >>> yourself), contains debug info. >> >> Ah, yes. Given a suitably large machine and enough time to do other stuff, >> that did the trick for me. Here's the result: >> >> http://cython.org/callgrind-pypy-nbody.png >> >> As you can see, make_ref() and Py_DecRef() combined account for almost 80% >> of the runtime. So the expected gain from optimising the PyObject handling >> is *huge*. >> >> The first thing that pops up from the graph is the different calls through >> generic_cpy_call(). That looks way to generic for something as performance >> critical as Py_DecRef(). >> >> Also, what's that recursive "stackwalk()" thing doing? > > Haha :) > > This is related to garbage collection - it scans the stack for GC pointers > to save them I think. We might think that it's a bit too much to do it for > every single call there. So, what's the plan of doing something about it? Stefan From fijall at gmail.com Thu Aug 23 11:17:12 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 23 Aug 2012 11:17:12 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: On Thu, Aug 23, 2012 at 11:11 AM, Stefan Behnel wrote: > Maciej Fijalkowski, 05.07.2012 14:50: >> On Thu, Jul 5, 2012 at 2:35 PM, Stefan Behnel wrote: >>> Maciej Fijalkowski, 05.07.2012 11:01: >>>> On Thu, Jul 5, 2012 at 10:26 AM, Amaury Forgeot d'Arc wrote: >>>>> 2012/7/5 Stefan Behnel >>>>>> Back to that question then: >>>>>> >>>>>>> Is there a way to get readable debugging symbols in a translated PyPy >>>>>>> that would tell me what is being executed? >>>>> >>>>> I fear that pypy standard distribution calls "strip" on the resulting >>>>> binary. >>>>> You could translate pypy yourself, I'm quite sure it contains debug info >>>>> already and it's quite easy to call "make debug" anyway. >>>> >>>> Default build (not the distribution or nightly, you have to trasnlate >>>> yourself), contains debug info. >>> >>> Ah, yes. Given a suitably large machine and enough time to do other stuff, >>> that did the trick for me. Here's the result: >>> >>> http://cython.org/callgrind-pypy-nbody.png >>> >>> As you can see, make_ref() and Py_DecRef() combined account for almost 80% >>> of the runtime. So the expected gain from optimising the PyObject handling >>> is *huge*. >>> >>> The first thing that pops up from the graph is the different calls through >>> generic_cpy_call(). That looks way to generic for something as performance >>> critical as Py_DecRef(). >>> >>> Also, what's that recursive "stackwalk()" thing doing? >> >> Haha :) >> >> This is related to garbage collection - it scans the stack for GC pointers >> to save them I think. We might think that it's a bit too much to do it for >> every single call there. > > So, what's the plan of doing something about it? > > Stefan I took a look at it and it seems it's not what I thought it is. It's just an intermediate call that saves stack roots and then calls the actual cpyext. I don't think the call itself is harmful, it just happens to be on the callstack (always) From stefan_ml at behnel.de Thu Aug 23 11:16:04 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 11:16:04 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Stefan Behnel, 03.07.2012 19:56: > Amaury Forgeot d'Arc, 03.07.2012 18:26: >> Maybe we should use a weak dictionary to cache the PyObject structure. >> This already exists for objects defined and created from C... > > That would be really helpful. In particular, it would solve one of the most > annoying problems that extensions currently have: even if you keep a Python > reference to an object, e.g. in a list, its PyObject structure will die > once the last C reference to it is gone. That is really hard to work around > in some cases. It's very common to keep e.g. a Python list (or set) of byte > strings and pass their char* buffer pointers into a C library. That doesn't > currently work with cpyext. Ok, so where would this have to be done? Is there a way to implement it generically in that ubiquitous space.wrap() kind of call (whatever that does internally), or would it have to be done explicitly whenever objects pass the boundary? Stefan From stefan_ml at behnel.de Thu Aug 23 11:25:21 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 11:25:21 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Maciej Fijalkowski, 23.08.2012 11:17: > On Thu, Aug 23, 2012 at 11:11 AM, Stefan Behnel wrote: >> Maciej Fijalkowski, 05.07.2012 14:50: >>> On Thu, Jul 5, 2012 at 2:35 PM, Stefan Behnel wrote: >>>> Maciej Fijalkowski, 05.07.2012 11:01: >>>>> On Thu, Jul 5, 2012 at 10:26 AM, Amaury Forgeot d'Arc wrote: >>>>>> 2012/7/5 Stefan Behnel >>>>>>> Back to that question then: >>>>>>> >>>>>>>> Is there a way to get readable debugging symbols in a translated PyPy >>>>>>>> that would tell me what is being executed? >>>>>> >>>>>> I fear that pypy standard distribution calls "strip" on the resulting >>>>>> binary. >>>>>> You could translate pypy yourself, I'm quite sure it contains debug info >>>>>> already and it's quite easy to call "make debug" anyway. >>>>> >>>>> Default build (not the distribution or nightly, you have to trasnlate >>>>> yourself), contains debug info. >>>> >>>> Ah, yes. Given a suitably large machine and enough time to do other stuff, >>>> that did the trick for me. Here's the result: >>>> >>>> http://cython.org/callgrind-pypy-nbody.png >>>> >>>> As you can see, make_ref() and Py_DecRef() combined account for almost 80% >>>> of the runtime. So the expected gain from optimising the PyObject handling >>>> is *huge*. >>>> >>>> The first thing that pops up from the graph is the different calls through >>>> generic_cpy_call(). That looks way to generic for something as performance >>>> critical as Py_DecRef(). >>>> >>>> Also, what's that recursive "stackwalk()" thing doing? >>> >>> Haha :) >>> >>> This is related to garbage collection - it scans the stack for GC pointers >>> to save them I think. We might think that it's a bit too much to do it for >>> every single call there. >> >> So, what's the plan of doing something about it? > > I took a look at it and it seems it's not what I thought it is. > > It's just an intermediate call that saves stack roots and then calls > the actual cpyext. I don't think the call itself is harmful, it just > happens to be on the callstack (always) Ah, ok - good to know. Then I think our next best bet is to cache the PyObject structs for PyPy objects using a weak-key dict. That will fix two problems at the same time: 1) prune excessive create-decref-dealloc cycles 2) keep the PyObject pointer valid as long as the PyPy object is alive, thus preventing crashes for code that expects an object reference in a list (for example) to be enough to keep the C object representation alive. Stefan From arigo at tunes.org Thu Aug 23 11:53:53 2012 From: arigo at tunes.org (Armin Rigo) Date: Thu, 23 Aug 2012 11:53:53 +0200 Subject: [pypy-dev] Hello World in Windows with MinGW In-Reply-To: <2CF0DBBEB4B88F4EBA2CF49732D993C3117C2F46@BY2PRD0511MB441.namprd05.prod.outlook.com> References: <2CF0DBBEB4B88F4EBA2CF49732D993C3117C2F46@BY2PRD0511MB441.namprd05.prod.outlook.com> Message-ID: Hi, On Fri, Aug 17, 2012 at 3:04 PM, Welch, Matt wrote: > Any help would be appreciated. Do I need a different environment? Is it not > MinGW/Msys friendly? Sorry, at least I don't know MinGW/Msys at all. The PyPy translation toolchain is tested on Linux, OS/X, and Windows MSVC, and there are people that try to make sure it also works on cygwin and FreeBSD. You need to either use one of these platforms, or send us fixes as patches. A bient?t, Armin. From dje.gcc at gmail.com Fri Aug 24 02:50:29 2012 From: dje.gcc at gmail.com (David Edelsohn) Date: Thu, 23 Aug 2012 20:50:29 -0400 Subject: [pypy-dev] PPC64 backend compile_framework_7_interior failure Message-ID: Armin, et al, compile_framework_7_interior in test_zrpy_gc fails for the PPC64 backend in a strange way. Identical problems happen on the public system and my private system, but GDB on the public system is not permitting hardware watchpoints, so it is more difficult to catch the cause of the failure. The testcase segfaults in a load following the instructions generated for getinteriorfield_gc: 0xfffb74e1958: li r3,0 0xfffb74e195c: li r0,24 0xfffb74e1960: mullw r0,r3,r0 0xfffb74e1964: addic r0,r0,16 0xfffb74e1968: ldx r3,r30,r0 ; $r30 + $r0 = 0xfffb75ea030 => 0xfffb74e196c: ld r3,8(r3) ; $r3 = 904 The ldx is the load from getinteriorfield_gc and the subsequent load fails. "904" appears to correspond to "n + i * 100 + ..." in the testcase for i = 9. If I set a watchpoint on the address, it initially is written during memcpy of a minor collection: #0 0x00000fffb7d0daa0 in .memcpy () from /lib64/power7/libc.so.6 #1 0x000000001000fb0c in .pypy_g_trace___trace_drag_out () #2 0x0000000010014288 in .pypy_g_MiniMarkGC_collect_oldrefs_to_nursery () #3 0x0000000010015358 in .pypy_g_MiniMarkGC_minor_collection.part.0 () #4 0x000000001001ce04 in .pypy_g_MiniMarkGC_collect () at which point (gdb) x/gx 0xfffb75ea030 0xfffb75ea030: 0x00000fffb776c170 (gdb) x/gd 0x00000fffb776c170 0xfffb776c170: 904 In other words, the location contains a pointer to another location containing the value "904". The value at that location later is overwritten in jitted code: 0xfffb74e0dec: li r28,0 0xfffb74e0df0: li r0,24 0xfffb74e0df4: mullw r0,r28,r0 0xfffb74e0df8: addic r0,r0,16 0xfffb74e0dfc: stdx r3,r30,r0 0xfffb74e0e14: li r4,16 0xfffb74e0e18: add r4,r3,r4 0xfffb74e0e1c: li r3,904 0xfffb74e0e20: std r3,0(r4) ; $r4 = 0xfffb75ea030 (gdb) print/x $r4 $6 = 0xfffb75ea030 $r3 is stored as part of a setinteriorfield_gc sequence. Subsequently $r3+16 is used as an address to store "904". So now that location instead of containing a pointer to the value "904" has been overwritten with the value "904", which fails when it later is accessed. I don't know if this flattening is intentional and the code reading the value was not updated to compensate for some sort of unboxing or one level of indirection has been lost. Any guidance to help debug this would be appreciated. Thanks, David From jason.microyea at gmail.com Fri Aug 24 08:58:03 2012 From: jason.microyea at gmail.com (jason ye) Date: Fri, 24 Aug 2012 14:58:03 +0800 Subject: [pypy-dev] win64 Message-ID: Hi Does anybody know the progress of the WIN64 port? I have a python program which require >4GB memory, expecting WIN64 PYPY Thanks Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Fri Aug 24 10:11:59 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 24 Aug 2012 10:11:59 +0200 Subject: [pypy-dev] PPC64 backend compile_framework_7_interior failure In-Reply-To: References: Message-ID: On Fri, Aug 24, 2012 at 2:50 AM, David Edelsohn wrote: > Armin, et al, > > compile_framework_7_interior in test_zrpy_gc fails for the PPC64 > backend in a strange way. Identical problems happen on the public > system and my private system, but GDB on the public system is not > permitting hardware watchpoints, so it is more difficult to catch the > cause of the failure. > > The testcase segfaults in a load following the instructions generated > for getinteriorfield_gc: > > 0xfffb74e1958: li r3,0 > 0xfffb74e195c: li r0,24 > 0xfffb74e1960: mullw r0,r3,r0 > 0xfffb74e1964: addic r0,r0,16 > 0xfffb74e1968: ldx r3,r30,r0 ; $r30 + $r0 = 0xfffb75ea030 > => 0xfffb74e196c: ld r3,8(r3) ; $r3 = 904 > > The ldx is the load from getinteriorfield_gc and the subsequent load fails. > > "904" appears to correspond to "n + i * 100 + ..." in the testcase for i = 9. > > If I set a watchpoint on the address, it initially is written during > memcpy of a minor collection: > > #0 0x00000fffb7d0daa0 in .memcpy () from /lib64/power7/libc.so.6 > #1 0x000000001000fb0c in .pypy_g_trace___trace_drag_out () > #2 0x0000000010014288 in .pypy_g_MiniMarkGC_collect_oldrefs_to_nursery () > #3 0x0000000010015358 in .pypy_g_MiniMarkGC_minor_collection.part.0 () > #4 0x000000001001ce04 in .pypy_g_MiniMarkGC_collect () > > at which point > > (gdb) x/gx 0xfffb75ea030 > 0xfffb75ea030: 0x00000fffb776c170 > (gdb) x/gd 0x00000fffb776c170 > 0xfffb776c170: 904 > > In other words, the location contains a pointer to another location > containing the value "904". > > The value at that location later is overwritten in jitted code: > > 0xfffb74e0dec: li r28,0 > 0xfffb74e0df0: li r0,24 > 0xfffb74e0df4: mullw r0,r28,r0 > 0xfffb74e0df8: addic r0,r0,16 > 0xfffb74e0dfc: stdx r3,r30,r0 > 0xfffb74e0e14: li r4,16 > 0xfffb74e0e18: add r4,r3,r4 > 0xfffb74e0e1c: li r3,904 > 0xfffb74e0e20: std r3,0(r4) ; $r4 = 0xfffb75ea030 > > (gdb) print/x $r4 > $6 = 0xfffb75ea030 > > $r3 is stored as part of a setinteriorfield_gc sequence. Subsequently > $r3+16 is used as an address to store "904". So now that location > instead of containing a pointer to the value "904" has been > overwritten with the value "904", which fails when it later is > accessed. > > I don't know if this flattening is intentional and the code reading > the value was not updated to compensate for some sort of unboxing or > one level of indirection has been lost. Any guidance to help debug > this would be appreciated. > > Thanks, David > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev Hi David. Can you get the traces dumped so we can see? If you can get the jitviewer running, it'll show you the assembler mixed with resops. If not, you can try to fish the resops yourself (they contain the actual offset inside assembler). Let me know if you need some help with that. From fijall at gmail.com Fri Aug 24 10:13:17 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 24 Aug 2012 10:13:17 +0200 Subject: [pypy-dev] win64 In-Reply-To: References: Message-ID: On Fri, Aug 24, 2012 at 8:58 AM, jason ye wrote: > Hi > > Does anybody know the progress of the WIN64 port? > > I have a python program which require >4GB memory, expecting WIN64 PYPY Hi Jason The Windows development port is stalled right now (nothing happened for the past X months). There are no immediate plans to resume working on this as of my knowledge, however, anyone on the mailing list who is planning to do so, please speak up. Cheers, fijal From fijall at gmail.com Fri Aug 24 14:46:02 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 24 Aug 2012 14:46:02 +0200 Subject: [pypy-dev] spectacular fail while upgrading tannit Message-ID: Hi I tried to upgrade tannit to a new Ubuntu LTS server. Unfortunately ubuntu forces people to upgrade grub to grub2, which also does not work in the partition setup that we have on tannit. The machine is lost (unbootable) and someone with physical access have to go there and fix it. Cheers, fijal From jacob at openend.se Fri Aug 24 15:04:38 2012 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Fri, 24 Aug 2012 15:04:38 +0200 Subject: [pypy-dev] spectacular fail while upgrading tannit In-Reply-To: References: Message-ID: <201208241504.39656.jacob@openend.se> Friday 24 August 2012 you wrote: > Hi > > I tried to upgrade tannit to a new Ubuntu LTS server. Unfortunately > ubuntu forces people to upgrade grub to grub2, which also does not > work in the partition setup that we have on tannit. The machine is > lost (unbootable) and someone with physical access have to go there > and fix it. > > Cheers, > fijal Since this is more than a matter of rebooting the machine this can be fixed at the earliest on Monday. Jacob From fijall at gmail.com Fri Aug 24 15:51:11 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 24 Aug 2012 15:51:11 +0200 Subject: [pypy-dev] spectacular fail while upgrading tannit In-Reply-To: <201208241504.39656.jacob@openend.se> References: <201208241504.39656.jacob@openend.se> Message-ID: On Fri, Aug 24, 2012 at 3:04 PM, Jacob Hall?n wrote: > Friday 24 August 2012 you wrote: >> Hi >> >> I tried to upgrade tannit to a new Ubuntu LTS server. Unfortunately >> ubuntu forces people to upgrade grub to grub2, which also does not >> work in the partition setup that we have on tannit. The machine is >> lost (unbootable) and someone with physical access have to go there >> and fix it. >> >> Cheers, >> fijal > > Since this is more than a matter of rebooting the machine this can be fixed at > the earliest on Monday. > > Jacob > Thanks to iko, this is up by now. Hopefully will survive until the proper fix on monday :) Cheers, fijal From arigo at tunes.org Fri Aug 24 19:39:38 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Aug 2012 19:39:38 +0200 Subject: [pypy-dev] PPC64 backend compile_framework_7_interior failure In-Reply-To: References: Message-ID: Hi David, >From your mail, "904" seems to be a typeid. It cannot be a variable computed as "n + 100 * i + ...", because it appears directly in assembler as "li r3, 904". It seems the problem is more subtle than that, reusing bogus memory and falling over a "904" that is a typeid when reading "16(r3)", which should not be possible. This problem might be caused by randomness hard to debug, or maybe precisely by what the test is really about (see the comment at the start): the write barrier doesn't work correctly. Such a write barrier should turn out as a COND_CALL_GC_WB resop. Another path of investigation would be to check that SETINTERIORFIELD_GC behaves correctly in all cases. Right now I see that test_runner.py only checks with all arguments being boxes, and not with combinations of boxes and consts. A bient?t, Armin. From tbaldridge at gmail.com Sat Aug 25 05:53:45 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Fri, 24 Aug 2012 22:53:45 -0500 Subject: [pypy-dev] "Don't know how to convert" error Message-ID: I hit this bug this evening while translating my code: [translation:ERROR] return self.call('simple_call', hop) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/lltypesystem/rpbc.py", line 245, in call [translation:ERROR] return hop.llops.convertvar(v_result, rresult, hop.r_result) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 933, in convertvar [translation:ERROR] (r_from, r_to)) [translation:ERROR] TyperError: don't know how to convert from to [translation:ERROR] .. (system.rt:60)PolymorphicFunc.invoke3 [translation:ERROR] .. block at 48 with 1 exits [translation:ERROR] .. v14 = simple_call(v13, a_0, b_0, c_0) What could cause something like this? I can seem to figure out how to even go debugging this. I dug into the stack trace a bit and poked around in the locals, but I can't seem to find any more information. Any ideas? Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sat Aug 25 10:09:21 2012 From: arigo at tunes.org (Armin Rigo) Date: Sat, 25 Aug 2012 10:09:21 +0200 Subject: [pypy-dev] "Don't know how to convert" error In-Reply-To: References: Message-ID: Hi Timothy, On Sat, Aug 25, 2012 at 5:53 AM, Timothy Baldridge wrote: > [translation:ERROR] TyperError: don't know how to convert from > to > [translation:ERROR] .. (system.rt:60)PolymorphicFunc.invoke3 > [translation:ERROR] .. block at 48 with 1 exits > [translation:ERROR] .. v14 = simple_call(v13, a_0, b_0, c_0) > > What could cause something like this? I can seem to figure out how to even > go debugging this. I dug into the stack trace a bit and poked around in the > locals, but I can't seem to find any more information. The problem is located in the system.rt module line 60, in PolymorphicFunc.invoke3(). It's probably a method call with three arguments called a, b, c. The error means there is some inconsistency between the type annotation (found "Void", maybe for the result of the call) and the actual RTyped call (which returns an instance of "system.core.Object"). It may be a bug of the ootype type system. That's all I can guess without inspecting the complete RPython sources (and not just the invoke3() function). A bient?t, Armin. From fijall at gmail.com Sat Aug 25 11:04:01 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 25 Aug 2012 11:04:01 +0200 Subject: [pypy-dev] [pypy-commit] pypy ppc-jit-backend: For getinteriorfield and setinteriorfield, if immediate offset is too In-Reply-To: <20120824185638.BB2D21C021F@cobra.cs.uni-duesseldorf.de> References: <20120824185638.BB2D21C021F@cobra.cs.uni-duesseldorf.de> Message-ID: Such fixes should come with tests in test_runner On Fri, Aug 24, 2012 at 8:56 PM, edelsohn wrote: > Author: edelsohn > Branch: ppc-jit-backend > Changeset: r56844:a4efe6ca0483 > Date: 2012-08-24 14:56 -0400 > http://bitbucket.org/pypy/pypy/changeset/a4efe6ca0483/ > > Log: For getinteriorfield and setinteriorfield, if immediate offset is > too large for instruction, load it into reg. > > diff --git a/pypy/jit/backend/ppc/regalloc.py b/pypy/jit/backend/ppc/regalloc.py > --- a/pypy/jit/backend/ppc/regalloc.py > +++ b/pypy/jit/backend/ppc/regalloc.py > @@ -751,7 +751,8 @@ > if _check_imm_arg(ofs): > ofs_loc = imm(ofs) > else: > - ofs_loc = self._ensure_value_is_boxed(ConstInt(ofs), args) > + ofs_loc = self.get_scratch_reg(INT, args) > + self.assembler.load(ofs_loc, imm(ofs)) > self.possibly_free_vars_for_op(op) > self.free_temp_vars() > result_loc = self.force_allocate_reg(op.result) > @@ -770,7 +771,8 @@ > if _check_imm_arg(ofs): > ofs_loc = imm(ofs) > else: > - ofs_loc = self._ensure_value_is_boxed(ConstInt(ofs), args) > + ofs_loc = self.get_scratch_reg(INT, args) > + self.assembler.load(ofs_loc, imm(ofs)) > return [base_loc, index_loc, value_loc, ofs_loc, imm(ofs), > imm(itemsize), imm(fieldsize)] > prepare_setinteriorfield_raw = prepare_setinteriorfield_gc > _______________________________________________ > pypy-commit mailing list > pypy-commit at python.org > http://mail.python.org/mailman/listinfo/pypy-commit From tbaldridge at gmail.com Sat Aug 25 15:26:48 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Sat, 25 Aug 2012 08:26:48 -0500 Subject: [pypy-dev] graph not in self.fixed_graphs Message-ID: I'm back ;-) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 176, in setup [translation:ERROR] self.add_source_for_attribute(name, source) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 193, in add_source_for_attribute [translation:ERROR] attrdef.mutated(cdef) # reflow from all read positions [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 109, in mutated [translation:ERROR] self.bookkeeper.annotator.reflowfromposition(position) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/annrpython.py", line 370, in reflowfromposition [translation:ERROR] self.reflowpendingblock(graph, block) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/annrpython.py", line 432, in reflowpendingblock [translation:ERROR] assert graph not in self.fixed_graphs [translation:ERROR] AssertionError The best I can infer is that perhaps a function is returning two types somewhere in my rpython code? If that were the case though, I'd assume that would blow up with a "SomeObject" error. I've never seen this error before, and it has me a bit baffled. I was able to debug it enough to find out where the code is that it is happening, but the code itself is nothing special: def first(self): return rt.first.invoke1(self) Of course, I have python tests for all this, and if I run my rpython code through CPython it runs without a hitch. What should I be doing to debug this sort of thing more on my own? Thanks for all the help so far... Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sat Aug 25 16:00:37 2012 From: arigo at tunes.org (Armin Rigo) Date: Sat, 25 Aug 2012 16:00:37 +0200 Subject: [pypy-dev] graph not in self.fixed_graphs In-Reply-To: References: Message-ID: Hi Timothy, On Sat, Aug 25, 2012 at 3:26 PM, Timothy Baldridge wrote: > ... > [translation:ERROR] assert graph not in self.fixed_graphs > [translation:ERROR] AssertionError For this case, we'd need to see the complete traceback. It's a situation of finding out late the existence of some extra attribute value. How late is shown near the start of the traceback. Which attribute it is is shown e.g. by the "name" and "source" local variables in the call "self.add_source_for_attribute(name, source)". The reason we see this extra attribute value is shown in the previous levels. A bient?t, Armin From tbaldridge at gmail.com Sat Aug 25 16:16:39 2012 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Sat, 25 Aug 2012 09:16:39 -0500 Subject: [pypy-dev] graph not in self.fixed_graphs In-Reply-To: References: Message-ID: Here's the complete traceback: [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "/home/tim/pypy/pypy/translator/goal/translate.py", line 308, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/tim/pypy/pypy/translator/driver.py", line 791, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/tim/pypy/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/tim/pypy/pypy/translator/driver.py", line 285, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/tim/pypy/pypy/translator/driver.py", line 371, in task_rtype_lltype [translation:ERROR] crash_on_first_typeerror=insist) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 211, in specialize [translation:ERROR] self.specialize_more_blocks() [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 254, in specialize_more_blocks [translation:ERROR] self.specialize_block(block) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 406, in specialize_block [translation:ERROR] self.translate_hl_to_ll(hop, varmapping) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 535, in translate_hl_to_ll [translation:ERROR] resultvar = hop.dispatch() [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 768, in dispatch [translation:ERROR] return translate_meth(self) [translation:ERROR] File "<329-codegen /home/tim/pypy/pypy/rpython/rtyper.py:610>", line 4, in translate_op_getattr [translation:ERROR] return r_arg1.rtype_getattr(hop) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/lltypesystem/rclass.py", line 555, in rtype_getattr [translation:ERROR] vinst, vattr = hop.inputargs(self, Void) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 799, in inputargs [translation:ERROR] vars.append(self.inputarg(converted_to[i], i)) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rtyper.py", line 780, in inputarg [translation:ERROR] return inputconst(converted_to, v.value) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rmodel.py", line 393, in inputconst [translation:ERROR] value = reqtype.convert_const(value) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rclass.py", line 335, in convert_const [translation:ERROR] return self.convert_const_exact(value) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rclass.py", line 344, in convert_const_exact [translation:ERROR] self.initialize_prebuilt_instance(value, self.classdef, result) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rclass.py", line 362, in initialize_prebuilt_instance [translation:ERROR] self.initialize_prebuilt_data(value, classdef, result) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/lltypesystem/rclass.py", line 452, in initialize_prebuilt_data [translation:ERROR] llattrvalue = r.convert_const(attrvalue) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/lltypesystem/rdict.py", line 246, in convert_const [translation:ERROR] llvalue = r_value.convert_const(dictvalue) [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rclass.py", line 322, in convert_const [translation:ERROR] classdef = bk.getuniqueclassdef(value.__class__) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/bookkeeper.py", line 281, in getuniqueclassdef [translation:ERROR] return desc.getuniqueclassdef() [translation:ERROR] File "/home/tim/pypy/pypy/annotation/description.py", line 560, in getuniqueclassdef [translation:ERROR] return self.getclassdef(None) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/description.py", line 546, in getclassdef [translation:ERROR] classdef.setup(classsources) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 176, in setup [translation:ERROR] self.add_source_for_attribute(name, source) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 193, in add_source_for_attribute [translation:ERROR] attrdef.mutated(cdef) # reflow from all read positions [translation:ERROR] File "/home/tim/pypy/pypy/annotation/classdef.py", line 109, in mutated [translation:ERROR] self.bookkeeper.annotator.reflowfromposition(position) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/annrpython.py", line 370, in reflowfromposition [translation:ERROR] self.reflowpendingblock(graph, block) [translation:ERROR] File "/home/tim/pypy/pypy/annotation/annrpython.py", line 432, in reflowpendingblock [translation:ERROR] assert graph not in self.fixed_graphs [translation:ERROR] AssertionError [translation] start debugger... > /home/tim/pypy/pypy/annotation/annrpython.py(432)reflowpendingblock() -> assert graph not in self.fixed_graphs (Pdb+) graph -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sat Aug 25 16:35:04 2012 From: arigo at tunes.org (Armin Rigo) Date: Sat, 25 Aug 2012 16:35:04 +0200 Subject: [pypy-dev] graph not in self.fixed_graphs In-Reply-To: References: Message-ID: Hi Timothy, On Sat, Aug 25, 2012 at 4:16 PM, Timothy Baldridge wrote: > [translation:ERROR] llvalue = r_value.convert_const(dictvalue) > [translation:ERROR] File "/home/tim/pypy/pypy/rpython/rclass.py", line > 322, in convert_const > [translation:ERROR] classdef = bk.getuniqueclassdef(value.__class__) > [translation:ERROR] File "/home/tim/pypy/pypy/annotation/bookkeeper.py", > line 281, in getuniqueclassdef > [translation:ERROR] return desc.getuniqueclassdef() > [translation:ERROR] File "/home/tim/pypy/pypy/annotation/description.py", > line 560, in getuniqueclassdef > [translation:ERROR] return self.getclassdef(None) Ok, so it's during regular rtyping. Here we see a prebuilt dictionary ("dictobj" in the first level above), containing in "dictvalue" a prebuilt instance of some class X. The problem is that it's the first time we ever see the class X. Doesn't really make sense to me. You may have to look in the callers a few extra levels up to figure out why we are seeing this dictionary; apparently from some attribute of some other prebuilt instance. The fact that this prebuilt instance allows us to reach (indirectly) class X, but class X was never seen during annotation, looks like a bug. A bient?t, Armin. From dje.gcc at gmail.com Sat Aug 25 21:01:20 2012 From: dje.gcc at gmail.com (David Edelsohn) Date: Sat, 25 Aug 2012 15:01:20 -0400 Subject: [pypy-dev] PPC64 backend compile_framework_7_interior failure In-Reply-To: References: Message-ID: I have a newer version of GDB installed in my account on the gcc110 that has hw watchpoint working. We can continue to add debugging capabilities to the PPC64 backend, but it is fairly clear how the ops correspond to assembler instructions. The overwriting occurs in the first call to setinteriorfield: ; function descriptor data 0xfffb74e0d18: fnmadd. f31,f31,f31,f31 0xfffb74e0d1c: .long 0xfff 0xfffb74e0d20: sthu r26,7712(r14) 0xfffb74e0d24: .long 0x0 0xfffb74e0d28: .long 0x0 0xfffb74e0d2c: .long 0x0 0xfffb74e0d30: .long 0x0 ; p13 = getfield_gc(ConstPtr(ptr12), descr=) 0xfffb74e0d34: lis r30,4131 0xfffb74e0d38: ori r30,r30,11712 0xfffb74e0d3c: ld r30,8(r30) ; guard_nonnull(p13, descr=) [i0, p13, p11, p10, p9, p8, p7, p6, p5, p4, p3, p2, p1] 0xfffb74e0d40: cmpdi r30,0 0xfffb74e0d44: beq 0xfffb74e1f60 ; i15 = int_add(i0, 0) 0xfffb74e0d48: addi r29,r3,0 ; i17 = int_add(i15, 1) 0xfffb74e0d4c: addi r29,r29,1 ; p18 = new(, descr=) ; malloc_cond (save r3, r4) 0xfffb74e0d50: std r3,-8(r31) 0xfffb74e0d54: std r4,-16(r31) ; load nursery_free_adr 0xfffb74e0d58: lis r3,4131 0xfffb74e0d5c: ori r3,r3,2864 0xfffb74e0d60: ld r3,0(r3) ; add size 0xfffb74e0d64: addi r4,r3,584 ; load nursery_top_adr 0xfffb74e0d68: lis r0,4131 0xfffb74e0d6c: ori r0,r0,2888 0xfffb74e0d70: ldx r0,0,r0 ; compare 0xfffb74e0d74: cmpld r4,r0 0xfffb74e0d78: ble 0xfffb74e0da0 ; _write_fail_index 0xfffb74e0d7c: li r0,2 0xfffb74e0d80: std r0,464(r31) ; malloc_slowpath 0xfffb74e0d84: lis r0,0 0xfffb74e0d88: ori r0,r0,4095 0xfffb74e0d8c: rldicr r0,r0,32,31 0xfffb74e0d90: oris r0,r0,46926 0xfffb74e0d94: ori r0,r0,1504 0xfffb74e0d98: mtctr r0 0xfffb74e0d9c: bctrl ; write to nursery_free_adr 0xfffb74e0da0: lis r0,4131 0xfffb74e0da4: ori r0,r0,2864 0xfffb74e0da8: stdx r4,0,r0 ; store typeid 0xfffb74e0dac: li r4,760 0xfffb74e0db0: std r4,0(r3) ; setfield_gc(p18, i17, descr=) 0xfffb74e0db4: std r29,8(r3) ; cond_call_gc_wb 0xfffb74e0db8: lbz r0,3(r30) 0xfffb74e0dbc: andi. r0,r0,1 0xfffb74e0dc0: beq 0xfffb74e0de4 0xfffb74e0dc4: mr r3,r30 0xfffb74e0dc8: lis r0,0 0xfffb74e0dcc: ori r0,r0,4095 0xfffb74e0dd0: rldicr r0,r0,32,31 0xfffb74e0dd4: oris r0,r0,46926 0xfffb74e0dd8: nop 0xfffb74e0ddc: mtctr r0 0xfffb74e0de0: bctrl ; setinteriorfield_gc(p13, 0, p18, descr=>) 0xfffb74e0de4: li r29,0 0xfffb74e0de8: mulli r0,r29,24 0xfffb74e0dec: addic r0,r0,16 0xfffb74e0df0: stdx r3,r30,r0 ; reload i0 0xfffb74e0df4: ld r29,-8(r31) ; i21 = int_add(i0, 0) 0xfffb74e0df8: addi r4,r29,0 ; i23 = int_add(i21, 2) 0xfffb74e0dfc: addi r4,r4,2 ; p24 = new(, descr=) 0xfffb74e0e00: addi r3,r3,16 ; store typeid 0xfffb74e0e04: li r28,760 0xfffb74e0e08: std r28,0(r3) ; XXX writes typeid @ offset 0 ; setfield_gc(p24, i23, descr=) => 0xfffb74e0e0c: std r4,8(r3) ; write field @ offset 8 ; cond_call_gc_wb 0xfffb74e0e10: lbz r0,3(r30) 0xfffb74e0e14: andi. r0,r0,1 0xfffb74e0e18: beq 0xfffb74e0e3c 0xfffb74e0e1c: mr r3,r30 0xfffb74e0e20: lis r0,0 0xfffb74e0e24: ori r0,r0,4095 0xfffb74e0e28: rldicr r0,r0,32,31 0xfffb74e0e2c: oris r0,r0,46926 0xfffb74e0e30: nop 0xfffb74e0e34: mtctr r0 0xfffb74e0e38: bctrl ; setinteriorfield_gc(p13, 0, p24, descr=>) 0xfffb74e0e3c: li r4,0 0xfffb74e0e40: mulli r0,r4,24 0xfffb74e0e44: addic r0,r0,24 0xfffb74e0e48: stdx r3,r30,r0 ; store pointer to object into interior field ... ; setinteriorfield_gc(p13, 9, p173, descr=>) 0xfffb74e176c: li r4,9 0xfffb74e1770: mulli r0,r4,24 0xfffb74e1774: addic r0,r0,32 0xfffb74e1778: stdx r3,r30,r0 ; p175 = getinteriorfield_gc(p13, 0, descr=>) 0xfffb74e177c: li r4,0 0xfffb74e1780: mulli r0,r4,24 0xfffb74e1784: addic r0,r0,16 0xfffb74e1788: ldx r4,r30,r0 ; i176 = getfield_gc(p175, descr=) => 0xfffb74e178c: ld r4,8(r4) ; XXX uses typeid as address ; i178 = int_add(i0, 0) 0xfffb74e1790: addi r28,r29,0 ; i180 = int_add(i178, 1) 0xfffb74e1794: addi r28,r28,1 ; i181 = int_eq(i176, i180) 0xfffb74e1798: cmpd r4,r28 0xfffb74e179c: crmove lt,eq 0xfffb74e17a0: mfcr r28 0xfffb74e17a4: rlwinm r28,r28,1,31,31 ; guard_true(i181, descr=) [i0, p13, p11, p10, p9, p8, p7, p6, p5, p4, p3, p2, p1] 0xfffb74e17a8: cmpdi r28,0 0xfffb74e17ac: beq 0xfffb74e1f94 ; p182 = getinteriorfield_gc(p13, 0, descr=>) 0xfffb74e17b0: li r28,0 0xfffb74e17b4: mulli r0,r28,24 0xfffb74e17b8: addic r0,r0,24 0xfffb74e17bc: ldx r28,r30,r0 ; i183 = getfield_gc(p182, descr=) 0xfffb74e17c0: ld r28,8(r28) It looks ilke something is conflicting with object allocation because the typeid of the second object created later is being loaded as the field from the first object. - David From arigo at tunes.org Sat Aug 25 23:19:38 2012 From: arigo at tunes.org (Armin Rigo) Date: Sat, 25 Aug 2012 23:19:38 +0200 Subject: [pypy-dev] PPC64 backend compile_framework_7_interior failure In-Reply-To: References: Message-ID: Hi all, For reference, we found the answer on IRC: On Sat, Aug 25, 2012 at 9:01 PM, David Edelsohn wrote: > ; cond_call_gc_wb > 0xfffb74e0e10: lbz r0,3(r30) > 0xfffb74e0e14: andi. r0,r0,1 > 0xfffb74e0e18: beq 0xfffb74e0e3c > 0xfffb74e0e1c: mr r3,r30 > 0xfffb74e0e20: lis r0,0 > 0xfffb74e0e24: ori r0,r0,4095 > 0xfffb74e0e28: rldicr r0,r0,32,31 > 0xfffb74e0e2c: oris r0,r0,46926 > 0xfffb74e0e30: nop > 0xfffb74e0e34: mtctr r0 > 0xfffb74e0e38: bctrl This code mangles the content of r3 (sometimes), which is supposed to be left intact. A bient?t, Armin. From markflorisson88 at gmail.com Mon Aug 27 15:09:51 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Mon, 27 Aug 2012 14:09:51 +0100 Subject: [pypy-dev] numpypy array expressions Message-ID: Hey, For this year's summer of code, and for my master dissertation, we created a project to compile array expressions efficiently (Dag was my mentor, CCed), which can be found here: https://github.com/markflorisson88/minivect , the thesis is under subdirectory 'thesis'. It's currently integrated in Cython, and can significantly outperform open source and commercial Fortran compilers for certain benchmarks (see the bench/graphs subdirectory on github, or the dissertation). For other benchmarks, it's mostly close, accounting for some runtime overhead. The purpose of the project is to be reusable, it's used by Cython, it's likely going to be integrated into Theano, it's partially integrated into numba at the moment. It can generate C code for static compilers, and uses llvmpy to do just-in-time specialization for runtime compilers (likely combined with some form of lazy evaluation). However, I think pypy has its own approach, and I think it relies on the JIT to evaluate the expressions (feel free to point me to documentation or source code)? But I don't imagine it optimizes for the cache hierarchy? So the question is, would there be an interest to use this project in PyPy? It's currently meant to be used as a git submodule, or to be included verbatim, but I also intend to make it distributable and installable. The project is designed so that adding new code generators should require minimal effort. But in the event this type of code translation is not compatible with pypy's approach, you could have a look at the techniques for optimizations, such as tiling for spatial locality and several of the optimizations described in the thesis and prototyped in code on github, such as SIMD transposes when mixing C and Fortran contiguous arrays, a tractable way to prove data independence for NumPy arrays to avoid array temporaries, optimizing broadcasting through loop-invariant code motion, and so forth. Anyway, it doesn't have all functionality yet, for instance it doesn't support reductions yet. But let us know if pypy would want to use it, and if so, how we could collaborate and make minivect compatible for the purposes of pypy. Mark From alex.eri at yandex.com Tue Aug 28 02:57:54 2012 From: alex.eri at yandex.com (Stepanov Aleksandr) Date: Tue, 28 Aug 2012 04:57:54 +0400 Subject: [pypy-dev] wsgi Message-ID: <273471346115474@web12g.yandex.ru> Are there any wsgi server for latest or latest stable pypy? uwsgi needs to be built with pypy-c and shared libpypy , but i failed to build pypy as described on uwsgi wiki after 5 hours of compiling. i have not find precompiled uwsgi_pypy. uwsgi_python does not works with pypy. gunicorn starts, but workers fall without traceback. have somebody recipe to join nginx and pypy? -- Best regards, Aleksandr. From pjenvey at underboss.org Tue Aug 28 04:04:11 2012 From: pjenvey at underboss.org (Philip Jenvey) Date: Mon, 27 Aug 2012 19:04:11 -0700 Subject: [pypy-dev] wsgi In-Reply-To: <273471346115474@web12g.yandex.ru> References: <273471346115474@web12g.yandex.ru> Message-ID: On Aug 27, 2012, at 5:57 PM, Stepanov Aleksandr wrote: > Are there any wsgi server for latest or latest stable pypy? > > uwsgi needs to be built with pypy-c and shared libpypy , but i failed to build pypy as described on uwsgi wiki after 5 hours of compiling. i have not find precompiled uwsgi_pypy. uwsgi_python does not works with pypy. > > gunicorn starts, but workers fall without traceback. > > have somebody recipe to join nginx and pypy? I thought gunicorn worked, though you don't get the full benefit of greenlets + pypy's JIT. Any of the pure python wsgi servers work, e.g. paste.httpserver, CherryPy's (cheroot), waitress. -- Philip Jenvey From roberto at unbit.it Tue Aug 28 07:40:37 2012 From: roberto at unbit.it (Roberto De Ioris) Date: Tue, 28 Aug 2012 07:40:37 +0200 Subject: [pypy-dev] wsgi In-Reply-To: <273471346115474@web12g.yandex.ru> References: <273471346115474@web12g.yandex.ru> Message-ID: Il giorno 28/ago/2012, alle ore 02:57, Stepanov Aleksandr ha scritto: > Are there any wsgi server for latest or latest stable pypy? > > uwsgi needs to be built with pypy-c and shared libpypy , but i failed to build pypy as described on uwsgi wiki after 5 hours of compiling. i have not find precompiled uwsgi_pypy. uwsgi_python does not works with pypy. > > gunicorn starts, but workers fall without traceback. > > have somebody recipe to join nginx and pypy? > libpypy-c -related patches are now in the official repository, you can build from that instead of the unbit-one that is no more updated (i need to fix the docs) -- Roberto De Ioris http://unbit.it JID: roberto at jabber.unbit.it From dmitrey15 at ukr.net Tue Aug 28 10:18:03 2012 From: dmitrey15 at ukr.net (Dmitrey) Date: Tue, 28 Aug 2012 11:18:03 +0300 Subject: [pypy-dev] [ANN] global nonlinear optimization constrained solver for PyPy Message-ID: <503C7EBB.6010205@ukr.net> Hi all, FYI global nonlinear optimization (GLP ) constrained solver *de* (differential evolution) from free OpenOpt framework ( http://openopt.org ) now works with PyPy, for more details see http://forum.openopt.org/viewtopic.php?id=615 ----------------- Regards, D. http://openopt.org/Dmitrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Aug 28 11:02:55 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 28 Aug 2012 11:02:55 +0200 Subject: [pypy-dev] [ANN] global nonlinear optimization constrained solver for PyPy In-Reply-To: <503C7EBB.6010205@ukr.net> References: <503C7EBB.6010205@ukr.net> Message-ID: On Tue, Aug 28, 2012 at 10:18 AM, Dmitrey wrote: > Hi all, > > FYI global nonlinear optimization (GLP) constrained solver de (differential > evolution) from free OpenOpt framework ( http://openopt.org ) now works with > PyPy, for more details see http://forum.openopt.org/viewtopic.php?id=615 > > ----------------- > Regards, D. > http://openopt.org/Dmitrey Thanks dmitrey, I'll get on the topic of indexing by int arrays at some point soon. Cheers, fijal From fijall at gmail.com Tue Aug 28 11:11:40 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 28 Aug 2012 11:11:40 +0200 Subject: [pypy-dev] wsgi In-Reply-To: <273471346115474@web12g.yandex.ru> References: <273471346115474@web12g.yandex.ru> Message-ID: On Tue, Aug 28, 2012 at 2:57 AM, Stepanov Aleksandr wrote: > Are there any wsgi server for latest or latest stable pypy? > > uwsgi needs to be built with pypy-c and shared libpypy , but i failed to build pypy as described on uwsgi wiki after 5 hours of compiling. i have not find precompiled uwsgi_pypy. uwsgi_python does not works with pypy. > > gunicorn starts, but workers fall without traceback. > > have somebody recipe to join nginx and pypy? > -- > Best regards, Aleksandr. Very likely your best bet is tornado as of now. It gets decent performance even. You can try twisted web for something slower, but probably more stable. Cheers, fijal From arigo at tunes.org Tue Aug 28 12:19:32 2012 From: arigo at tunes.org (Armin Rigo) Date: Tue, 28 Aug 2012 12:19:32 +0200 Subject: [pypy-dev] numpypy array expressions In-Reply-To: References: Message-ID: Hi Mark, On Mon, Aug 27, 2012 at 3:09 PM, mark florisson wrote: > For this year's summer of code, and for my master dissertation, we > created a project to compile array expressions efficiently (Dag was my > mentor, CCed), which can be found here: > https://github.com/markflorisson88/minivect , the thesis is under > subdirectory 'thesis'. Thanks for mentioning it. It is a rather different approach than the one we are playing with in PyPy. It looks like a project that is independent of CPython-vs-PyPy: doing some computation on AST trees and producing some compiled form, possibly with llvm. This works independently on the underlying interpreter, but makes some unchecked assumptions e.g. on the exact Python types involved in the expression. By contrast, the approach in PyPy is at a different level, based on JITting by automatic transformation of the interpreter (like we do to run Python). The idea is to automatically transform the part of numpypy that interprets numpy expressions, to get a JIT out of it; and then the result is that at run-time the evaluation of numpy expressions is JITted. E.g. there is no AST anywhere in our approach. It integrates naturally the JITted code produced out of Python and the JITted code out of the numpypy parts. Someone can feel free to try to plug your approach into PyPy --- I am not saying no, but I am saying that it would be unlikely that you could reuse any of the infrastructure that we already have. Moreover, if it does not contains a lot of tests, none of your code is any use for us --- this point is the way we work, but also the only sane way to write anything in RPython, because otherwise the 45-minutes compilation cycle would kill us. As for reusing some ideas, maybe, but it's still a long way off for PyPy. Our goal is to first be as compatible as possible, and completely transparent to the user, whatever his numpy source code does. A bient?t, Armin. From dynamicgl at gmail.com Tue Aug 28 12:43:21 2012 From: dynamicgl at gmail.com (gelin yan) Date: Tue, 28 Aug 2012 18:43:21 +0800 Subject: [pypy-dev] wsgi In-Reply-To: References: <273471346115474@web12g.yandex.ru> Message-ID: On Tue, Aug 28, 2012 at 5:11 PM, Maciej Fijalkowski wrote: > On Tue, Aug 28, 2012 at 2:57 AM, Stepanov Aleksandr > wrote: > > Are there any wsgi server for latest or latest stable pypy? > > > > uwsgi needs to be built with pypy-c and shared libpypy , but i failed to > build pypy as described on uwsgi wiki after 5 hours of compiling. i have > not find precompiled uwsgi_pypy. uwsgi_python does not works with pypy. > > > > gunicorn starts, but workers fall without traceback. > > > > have somebody recipe to join nginx and pypy? > > -- > > Best regards, Aleksandr. > > Very likely your best bet is tornado as of now. It gets decent > performance even. You can try twisted web for something slower, but > probably more stable. > > Cheers, > fijal > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > Hi You can try cyclone (http://cyclone.io) it is twisted based. I ran some tests on ubuntu with pypy, The performance was quite impressive so I plan to use it with pypy for production purpose. -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Tue Aug 28 12:44:04 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 28 Aug 2012 11:44:04 +0100 Subject: [pypy-dev] numpypy array expressions In-Reply-To: References: Message-ID: Hey Armin, On 28 August 2012 11:19, Armin Rigo wrote: > Hi Mark, > > On Mon, Aug 27, 2012 at 3:09 PM, mark florisson > wrote: >> For this year's summer of code, and for my master dissertation, we >> created a project to compile array expressions efficiently (Dag was my >> mentor, CCed), which can be found here: >> https://github.com/markflorisson88/minivect , the thesis is under >> subdirectory 'thesis'. > > Thanks for mentioning it. It is a rather different approach than the > one we are playing with in PyPy. It looks like a project that is > independent of CPython-vs-PyPy: doing some computation on AST trees > and producing some compiled form, possibly with llvm. This works > independently on the underlying interpreter, but makes some unchecked > assumptions e.g. on the exact Python types involved in the expression. Indeed, it is a low-level compiler, things like type checking (other than numeric promotion) should be performed by the front-end (in this case a lazy evaluation of numpy). > By contrast, the approach in PyPy is at a different level, based on > JITting by automatic transformation of the interpreter (like we do to > run Python). The idea is to automatically transform the part of > numpypy that interprets numpy expressions, to get a JIT out of it; and > then the result is that at run-time the evaluation of numpy > expressions is JITted. E.g. there is no AST anywhere in our approach. > It integrates naturally the JITted code produced out of Python and > the JITted code out of the numpypy parts. Thanks for the clarification, I thought that was the case. > Someone can feel free to try to plug your approach into PyPy --- I am > not saying no, but I am saying that it would be unlikely that you > could reuse any of the infrastructure that we already have. Understandable. How it would work for a lazy evaluating numpy, or for projects like Theano, is that they convert their expression graph at runtime to minivect after performing all sanity checks/optimizations, to generate native code. If the code generation part is already figured out, then there may not be too much sense in reusing the project. > Moreover, if it does not contains a lot of tests, none of your code is > any use for us --- this point is the way we work, but also the only > sane way to write anything in RPython, because otherwise the > 45-minutes compilation cycle would kill us. Indeed, most of the tests are actually system tests part of Cython: https://github.com/markflorisson88/cython/tree/_array_expressions/tests/array_expressions . Writing a more comprehensive test suite as part of minivect is on the todo list. > As for reusing some ideas, maybe, but it's still a long way off for > PyPy. Our goal is to first be as compatible as possible, and > completely transparent to the user, whatever his numpy source code > does. > That should obviously have priority :) In any case, if it at some point does reach such a stage and you're shooting for more performance, you know where to find us. Rewriting all optimizations may be some work, but then, compared to implementing numpy it'll be a piece of cake :) > A bient?t, > > Armin. From stefan_ml at behnel.de Wed Aug 29 22:29:25 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 29 Aug 2012 22:29:25 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Stefan Behnel, 23.08.2012 11:16: > Stefan Behnel, 03.07.2012 19:56: >> Amaury Forgeot d'Arc, 03.07.2012 18:26: >>> Maybe we should use a weak dictionary to cache the PyObject structure. >>> This already exists for objects defined and created from C... >> >> That would be really helpful. In particular, it would solve one of the most >> annoying problems that extensions currently have: even if you keep a Python >> reference to an object, e.g. in a list, its PyObject structure will die >> once the last C reference to it is gone. That is really hard to work around >> in some cases. It's very common to keep e.g. a Python list (or set) of byte >> strings and pass their char* buffer pointers into a C library. That doesn't >> currently work with cpyext. > > Ok, so where would this have to be done? Is there a way to implement it > generically in that ubiquitous space.wrap() kind of call (whatever that > does internally), or would it have to be done explicitly whenever objects > pass the boundary? Any comments? I think this would solve a lot of problems. Stefan From arigo at tunes.org Thu Aug 30 09:10:03 2012 From: arigo at tunes.org (Armin Rigo) Date: Thu, 30 Aug 2012 09:10:03 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Hi Stefan, On Wed, Aug 29, 2012 at 10:29 PM, Stefan Behnel wrote: >> Ok, so where would this have to be done? Is there a way to implement it >> generically in that ubiquitous space.wrap() kind of call (whatever that >> does internally), or would it have to be done explicitly whenever objects >> pass the boundary? > > Any comments? I think this would solve a lot of problems. You need to do it at the level of the boundary --- space.wrap() is at a different level. We can't slow it down by doing dictionary lookups there, because it would be a big hit of performance for the whole PyPy. Fwiw I'm now thinking about a possible rewrite of cpyext, based on (an extension of) CFFI that would let us declare a C API and implement it in Python. This would mean basically moving cpyext to a regular app-level module. Doing so in general would be useful to write C extensions or wrappers for existing libraries: they would not necessarily be using the CPython C API but instead any custom C API that is most suitable for that particular extension. For Cython, it would mean that you would be free to invent whatever API is most suitable --- starting, I'm sure, with some subset of the CPython C API, but possibly evolving over time towards something that offers better performance in the context of PyPy. A bient?t, Armin. From stefan_ml at behnel.de Thu Aug 30 10:06:53 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 30 Aug 2012 10:06:53 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Hi Armin, Armin Rigo, 30.08.2012 09:10: > On Wed, Aug 29, 2012 at 10:29 PM, Stefan Behnel wrote: >>> Ok, so where would this have to be done? Is there a way to implement it >>> generically in that ubiquitous space.wrap() kind of call (whatever that >>> does internally), or would it have to be done explicitly whenever objects >>> pass the boundary? >> >> Any comments? I think this would solve a lot of problems. > > You need to do it at the level of the boundary --- space.wrap() is at > a different level. We can't slow it down by doing dictionary lookups > there, because it would be a big hit of performance for the whole > PyPy. Understood. This may not even be that hard to do, but it's unlikely that I'll give this a try myself. The turn-over times of building PyPy for an hour or so on a remote machine (RAM!) for most non-trivial changes are just way too long to make friends with the code base. Even waiting for minutes to get an interpreted PyPy in a state where it can start building (let alone running) Cython extensions is way beyond the seconds in my usual work flow. > Fwiw I'm now thinking about a possible rewrite of cpyext, based on (an > extension of) CFFI that would let us declare a C API and implement it > in Python. This would mean basically moving cpyext to a regular > app-level module. Doing so in general would be useful to write C > extensions or wrappers for existing libraries: they would not > necessarily be using the CPython C API but instead any custom C API > that is most suitable for that particular extension. For Cython, it > would mean that you would be free to invent whatever API is most > suitable --- starting, I'm sure, with some subset of the CPython C > API, but possibly evolving over time towards something that offers > better performance in the context of PyPy. Assuming that's doable in a reasonable time frame (including proper JITting of the API implementation to make it reasonably fast), that sounds like a good idea. Cython already uses quite a bit of special casing for cpyext by now, so replacing at least some of the more involved code blocks with a more direct C-API call would be nice. Actually the subset of the CPython C-API that Cython uses is not even all that large. Instead, we inline and special case a lot of cases that CPython needs to broaden its API for and tend to use the most low-level functions only. So Cython should prove a suitable target for such an API implementation. Funny enough, I had to change many low-level calls in CPython (PyTuple_*) to high-level calls in cpyext (PySequence_*), both because they are substantially faster (less useless error testing) and less likely to crash (no borrowed reference handling). It also sounds like this approach would actually enable a workable test work-flow for people like me. Stefan From arigo at tunes.org Thu Aug 30 10:28:30 2012 From: arigo at tunes.org (Armin Rigo) Date: Thu, 30 Aug 2012 10:28:30 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Hi Stefan, On Thu, Aug 30, 2012 at 10:06 AM, Stefan Behnel wrote: > This may not even be that hard to do, but it's unlikely that I'll give this > a try myself. The turn-over times of building PyPy for an hour or so on a > remote machine (RAM!) for most non-trivial changes are just way too long to > make friends with the code base. Even waiting for minutes to get an > interpreted PyPy in a state where it can start building (let alone running) > Cython extensions is way beyond the seconds in my usual work flow. This is a blocker, and one that the plan towards CFFI should completely fix. It's going to be so much more fun to write regular Python code and try it immediately :-) rather than hack at the interpreter and have to wait minutes for tests to run or an hour for a full pypy build. A bient?t, Armin. From janpeterr at freenet.de Thu Aug 30 12:45:10 2012 From: janpeterr at freenet.de (Jan Riechers) Date: Thu, 30 Aug 2012 13:45:10 +0300 Subject: [pypy-dev] pymongo compatibility broken with version 2.3 Message-ID: <503F4436.3010703@freenet.de> Hello, I know this is not particular subject for this mailing list, but this info might help someone else using pymongo for mongodb with pypy. --- Latest release of the mongodb driver, pymongo (version 2.3, off 29th, August) does not work anymore with Pypy 1.9 and fails to start up with something like this: " (env) D:\gg_webdev>pypy dbskin.py Fatal Python error: PyThreadState_Get: no current thread This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. " But from my code side, there is no greenlet or similar in usage. The problem just arrived when upgrading the driver. So I'm not sure whats the concrete problem with this new version - also after consulting the changelog of the driver ( https://jira.mongodb.org/browse/PYTHON/fixforversion/11146 ) there is nothing mentioned about threading or anything alike that - but it's definitely the culprid. Nevertheless, version 2.2.1 (pymongo-2.2.1_-py2.7.egg) does work like a charm. Jan From fijall at gmail.com Thu Aug 30 14:02:33 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 30 Aug 2012 14:02:33 +0200 Subject: [pypy-dev] pymongo compatibility broken with version 2.3 In-Reply-To: <503F4436.3010703@freenet.de> References: <503F4436.3010703@freenet.de> Message-ID: On Thu, Aug 30, 2012 at 12:45 PM, Jan Riechers wrote: > Hello, > > I know this is not particular subject for this mailing list, but this info > might help someone else using pymongo for mongodb with pypy. > > --- > > Latest release of the mongodb driver, pymongo (version 2.3, off 29th, > August) does not work anymore with Pypy 1.9 and fails to start up with > something like this: > > " > (env) D:\gg_webdev>pypy dbskin.py > Fatal Python error: PyThreadState_Get: no current thread > > This application has requested the Runtime to terminate it in an unusual > way. > Please contact the application's support team for more information. > " > > But from my code side, there is no greenlet or similar in usage. The problem > just arrived when upgrading the driver. > > So I'm not sure whats the concrete problem with this new version - also > after consulting the changelog of the driver ( > https://jira.mongodb.org/browse/PYTHON/fixforversion/11146 ) there is > nothing mentioned about threading or anything alike that - but it's > definitely the culprid. > > Nevertheless, version 2.2.1 (pymongo-2.2.1_-py2.7.egg) does work like a > charm. > > Jan > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev It's clearly doing something with CPython C API. Is it the default behavior? Cheers, fijal From amauryfa at gmail.com Thu Aug 30 14:25:49 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 30 Aug 2012 14:25:49 +0200 Subject: [pypy-dev] pymongo compatibility broken with version 2.3 In-Reply-To: <503F4436.3010703@freenet.de> References: <503F4436.3010703@freenet.de> Message-ID: 2012/8/30 Jan Riechers : > (env) D:\gg_webdev>pypy dbskin.py > Fatal Python error: PyThreadState_Get: no current thread This error message does not appear anywhere in pypy code, it certainly comes from CPython... This message is typical of an extension module that was not recompiled for pypy; it still links with python27.dll, and the call to PyThreadState_Get() jumps into an uninitialized CPython interpreter... Extension modules need to be recompiled for pypy. -- Amaury Forgeot d'Arc From janpeterr at freenet.de Thu Aug 30 17:42:48 2012 From: janpeterr at freenet.de (Jan Riechers) Date: Thu, 30 Aug 2012 18:42:48 +0300 Subject: [pypy-dev] pymongo compatibility broken with version 2.3 In-Reply-To: <503F7388.3010708@freenet.de> References: <503F7388.3010708@freenet.de> Message-ID: <503F89F8.1040701@freenet.de> On 30.08.2012 15:25, Amaury Forgeot d'Arc wrote: > 2012/8/30 Jan Riechers : >> (env) D:\gg_webdev>pypy dbskin.py >> Fatal Python error: PyThreadState_Get: no current thread > > This error message does not appear anywhere in pypy code, > it certainly comes from CPython... > > This message is typical of an extension module that was not recompiled for pypy; > it still links with python27.dll, and the call to PyThreadState_Get() jumps into > an uninitialized CPython interpreter... > > Extension modules need to be recompiled for pypy. > Thank you both for this explanation, and sorry for any confusion. I installed by bad habit using easy_install instead of setup.py install procedure from the github source. All good by now, 2.3 works fine! And sorry Amaury for the direct email response. Jan From wiktor8010 at o2.pl Thu Aug 30 21:43:31 2012 From: wiktor8010 at o2.pl (=?UTF-8?Q?Wiktor_Mizdal?=) Date: Thu, 30 Aug 2012 21:43:31 +0200 Subject: [pypy-dev] =?utf-8?q?pypy_android?= Message-ID: <31ea3354.7b1b5bb5.503fc263.134e1@o2.pl> Hi, does Pypy for Android require root Android version? Wiktor Mizdal From fijall at gmail.com Thu Aug 30 22:01:44 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 30 Aug 2012 22:01:44 +0200 Subject: [pypy-dev] pypy android In-Reply-To: <31ea3354.7b1b5bb5.503fc263.134e1@o2.pl> References: <31ea3354.7b1b5bb5.503fc263.134e1@o2.pl> Message-ID: On Thu, Aug 30, 2012 at 9:43 PM, Wiktor Mizdal wrote: > Hi, > > does Pypy for Android require root Android version? > > > Wiktor Mizdal Is there such a thing as pypy for android in the first place? From stefan_ml at behnel.de Fri Aug 31 09:22:26 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 31 Aug 2012 09:22:26 +0200 Subject: [pypy-dev] nightly builds require OpenSSL 1.0 ? Message-ID: Hi, my nightly fetch-PyPy build job stopped working two days ago because the nightly builds now require libssl 1.0.0. It's not currently available on our integration test server. Is that dependency really necessary or was it just an accidental side effect of a system update? Stefan From fijall at gmail.com Fri Aug 31 09:27:25 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 31 Aug 2012 09:27:25 +0200 Subject: [pypy-dev] nightly builds require OpenSSL 1.0 ? In-Reply-To: References: Message-ID: On Fri, Aug 31, 2012 at 9:22 AM, Stefan Behnel wrote: > Hi, > > my nightly fetch-PyPy build job stopped working two days ago because the > nightly builds now require libssl 1.0.0. It's not currently available on > our integration test server. > > Is that dependency really necessary or was it just an accidental side > effect of a system update? > > Stefan This was an effect of a system update. Binary compatibility on linux is hard :/ Cheers, fijal From arigo at tunes.org Fri Aug 31 09:31:14 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 31 Aug 2012 09:31:14 +0200 Subject: [pypy-dev] pypy android In-Reply-To: References: <31ea3354.7b1b5bb5.503fc263.134e1@o2.pl> Message-ID: Hi Maciej, On Thu, Aug 30, 2012 at 10:01 PM, Maciej Fijalkowski wrote: > Is there such a thing as pypy for android in the first place? I think that the ARM JIT backend supports Android, at least if you look at the version numbers of the processors. I don't know if this has been tested so far. A bient?t, Armin. From david.schneider at picle.org Fri Aug 31 10:10:51 2012 From: david.schneider at picle.org (David Schneider) Date: Fri, 31 Aug 2012 10:10:51 +0200 Subject: [pypy-dev] pypy android In-Reply-To: References: <31ea3354.7b1b5bb5.503fc263.134e1@o2.pl> Message-ID: <02867793-6EC0-427F-98B3-31A782C1A788@picle.org> Hi, > On Thu, Aug 30, 2012 at 10:01 PM, Maciej Fijalkowski wrote: >> Is there such a thing as pypy for android in the first place? To my knowledge there isn't really a PyPy for android so far. > > I think that the ARM JIT backend supports Android, at least if you > look at the version numbers of the processors. I don't know if this > has been tested so far. Based on the requirements PyPy could in Theory work on Android, it is Linux based, allows JIT compilation, the commonly used hardware matches what the backend supports. But AFAIK android uses a different libc, called bionic, and probably some other custom libraries making it a bit harder, but probably not impossible to cross translate for Android as we do for ARM/Linux using a chroot/qemu environment. David From arigo at tunes.org Fri Aug 31 10:15:29 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 31 Aug 2012 10:15:29 +0200 Subject: [pypy-dev] nightly builds require OpenSSL 1.0 ? In-Reply-To: References: Message-ID: Hi, On Fri, Aug 31, 2012 at 9:27 AM, Maciej Fijalkowski wrote: > This was an effect of a system update. Binary compatibility on linux is hard :/ I hacked *yet again another time* to link openssl statically in the binary, like it should have been before. A bient?t, Armin. From wiktor8010 at o2.pl Fri Aug 31 10:35:12 2012 From: wiktor8010 at o2.pl (=?UTF-8?Q?Wiktor_Mizdal?=) Date: Fri, 31 Aug 2012 10:35:12 +0200 Subject: [pypy-dev] =?utf-8?q?pypy_android?= Message-ID: <3cef4ff1.78b93f9.50407740.f0c43@o2.pl> Hi, In article "Almost There - PyPy's ARM Backend" on Pypy blog, in "The incomplete list of open topics:" is point "Build a version that runs on Android". Will somebody work on it? Wiktor Mizdal From carlosmf.pt at gmail.com Fri Aug 31 11:45:01 2012 From: carlosmf.pt at gmail.com (Carlos Ferreira) Date: Fri, 31 Aug 2012 10:45:01 +0100 Subject: [pypy-dev] Developer selection for Py3k In-Reply-To: References: <4FECE3EC.7030000@gmail.com> Message-ID: I would like to know, how can I contribute for this project :) I have some experience with C/C++ and Python, but the main reason would be, to learn even more, specially things that aren't usually found in books :) On 23 August 2012 01:32, Philip Jenvey wrote: > > On Jun 29, 2012, at 12:39 PM, Philip Jenvey wrote: > > > > > On Jun 28, 2012, at 4:08 PM, Antonio Cuni wrote: > > > >> Hi all, > >> > >> as you probably know, the Py3k [1] proposal is getting funded thanks to > our > >> generous donors. > >> > >> During the first round, three of use were selected: me, Alex Gaynor and > >> Benjamin Peterson. However, due to unforeseen unavailability of Alex > and > >> Benjamin, we are now looking for one more developer to help with the > py3k work. > >> > >> If you are interested in getting paid to work on the Py3k proposal, > please > >> apply by replying to this email. > >> > >> To be applicable you need to be an experienced PyPy developer, > preferably with > >> a previous experience on the Python Interpreter part of PyPy. > > > > If it was a few months ago I'd be up for this, unfortunately I'm not > available now. I'll let you know if that changes. > > I now have some free cycles available for Py3k work =] > > -- > Philip Jenvey > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -- Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf at av.it.pt University of Aveiro E-mail -> cmf at ua.pt MSN Contact -> carlosmf.pt at gmail.com Skype & GTalk -> carlosmf.pt at gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira -------------- next part -------------- An HTML attachment was scrubbed... URL: From santagada at gmail.com Fri Aug 31 14:46:46 2012 From: santagada at gmail.com (Leonardo Santagada) Date: Fri, 31 Aug 2012 09:46:46 -0300 Subject: [pypy-dev] pypy android In-Reply-To: <02867793-6EC0-427F-98B3-31A782C1A788@picle.org> References: <31ea3354.7b1b5bb5.503fc263.134e1@o2.pl> <02867793-6EC0-427F-98B3-31A782C1A788@picle.org> Message-ID: On Fri, Aug 31, 2012 at 5:10 AM, David Schneider wrote: > > > Based on the requirements PyPy could in Theory work on Android, it is Linux based, allows JIT compilation, the commonly used hardware matches what the backend supports. But AFAIK android uses a different libc, called bionic, and probably some other custom libraries making it a bit harder, but probably not impossible to cross translate for Android as we do for ARM/Linux using a chroot/qemu environment. what you would probably use is the android NDK http://developer.android.com/tools/sdk/ndk/index.html -- Leonardo Santagada From cfbolz at gmx.de Fri Aug 31 14:54:19 2012 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 31 Aug 2012 14:54:19 +0200 Subject: [pypy-dev] pypy android In-Reply-To: <3cef4ff1.78b93f9.50407740.f0c43@o2.pl> References: <3cef4ff1.78b93f9.50407740.f0c43@o2.pl> Message-ID: <5040B3FB.4070209@gmx.de> On 08/31/2012 10:35 AM, Wiktor Mizdal wrote: > Hi, > > In article "Almost There - PyPy's ARM Backend" on Pypy blog, in "The > incomplete list of open topics:" is point "Build a version that runs on > Android". Will somebody work on it? Nobody that we know of is working on this, or plans to. Contributions are welcome. Cheers, Carl Friedrich From arigo at tunes.org Fri Aug 31 15:53:24 2012 From: arigo at tunes.org (Armin Rigo) Date: Fri, 31 Aug 2012 15:53:24 +0200 Subject: [pypy-dev] cpyext performance In-Reply-To: References: Message-ID: Hi, On Thu, Aug 30, 2012 at 10:28 AM, Armin Rigo wrote: > This is a blocker, and one that the plan towards CFFI should > completely fix. A quick but successful attempt can be found in cffi/demo/{api,pyobj}.py. api.py is a bit of a hack but lets you declare functions with the decorator @ffi.pyexport("C function signature"). This gives you Python-implemented functions that you can call back from C. They are available from the C code passed into verify(). (A possible extension of this idea would be to choose the precise name and C-level interface that we want to give to the .so; with only a few extra hacks this could let the user write pure Python CFFI code to implement a .so that can be used in the system without knowledge that it is written in Python in the first place, e.g. as a plug-in to some existing program.) pyobj.py is a quick attempt at exposing the C API I described earlier, with "object descriptors". So far the API is only sufficient for the demo in this file, which is a C function that iterates over a list and computes the sum of its items. In the first version it assumes all items are numbers that fit in an "int". In the second version it really calls the Python addition, and so supports various kinds of objects. Note how the whole API is basically defined and implemented as regular Python code. These examples should work well on both CPython and PyPy. A bient?t, Armin.