From p.giarrusso at gmail.com Fri Oct 1 00:11:17 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Fri, 1 Oct 2010 00:11:17 +0200 Subject: [pypy-dev] Question on the future of RPython In-Reply-To: References: <525425.61205.qm@web53702.mail.re2.yahoo.com> <827221.6704.qm@web53704.mail.re2.yahoo.com> <1285616691.5954.34.camel@localhost> <201009280157.17264.jacob@openend.se> <1285634613.5954.112.camel@localhost> <1285703348.5276.56.camel@localhost> <1285809701.5129.150.camel@localhost> Message-ID: On Thu, Sep 30, 2010 at 13:01, Armin Rigo wrote: > Hi Paolo, > On Thu, Sep 30, 2010 at 8:33 AM, Paolo Giarrusso wrote: >> My proposal, here, would be a "virtual guard", (...) > Yes, this proposal makes sense. ?It's an optimization that is > definitely done in regular JITs, and we have a "to-do" task about it > in http://codespeak.net/svn/pypy/extradoc/planning/jit.txt (where they > are called "out-of-line guards"). I see - but while I know of what for instance Java does (I had an example), here one can reuse JITted traces (up to some point) rather than just throw everything out (except maybe profiling data), add the new code path and redo all optimizations when recompiling it. I don't see how a method-at-a-time JITs could reuse more than that, and my class notes just state that code can be thrown out. My proposal is that here, also the recorded traces could be reused (if they are stored or can be recovered), exactly because one just adds a different code path and tracing JITs reason in terms of single paths. Ideally, the same trace could be recompiled (this time with a guard) the next time it is entered, without using the trace-recording interpreter, nor waiting again that it is executed N times with N > compileThreshold. Potentially, in a trace-stitching JIT like PyPy, one could even just prepend a guard to the compiled assembly/binary code (if that's position-independent, like x86-64, or can be relocated). Of course, all of this is dependent on the representation used for the "code/trace caches", because I'm not sure all needed data is kept - and this shouldn't increase too much storage requirements, especially given that invalidations might not be very frequent. So, do you have pointers in the code? Do you prefer me to ask by chat? I just checked out sources on my new dev machine (lack of disk space was one reason why I never did it). Is the History class in pypy/jit/metainterp/history.py what represents a trace? Grepping for "trace" in that folder didn't help a lot. BTW, I am not used to bother authors by asking questions when reading sources; but as far as I saw, it is considered "socially acceptable" here, isn't it? Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From arigo at tunes.org Fri Oct 1 11:31:30 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 1 Oct 2010 11:31:30 +0200 Subject: [pypy-dev] =?iso-8859-1?q?Next_sprint=3A_D=FCsseldorf=2C_end_of_O?= =?iso-8859-1?q?ctober?= Message-ID: Hi all! D?sseldorf PyPy sprint October 25th-31st 2010 ===================================================== The next PyPy sprint will be held in the Computer Science department of Heinrich-Heine Universit?t D?sseldorf from the 25th to the 31st of October 2010. This is a fully public sprint, everyone is welcome to join us. Topics and goals ---------------- Open. (to be defined more precisely) Location -------- The sprint will take place in a seminar room of the computer science department. It is in the building 25.12 of the university campus. For travel instructions see http://stups.cs.uni-duesseldorf.de/anreise/esbahn.php Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2010/people.txt From arigo at tunes.org Fri Oct 1 12:02:49 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 1 Oct 2010 12:02:49 +0200 Subject: [pypy-dev] Question on the future of RPython In-Reply-To: References: <525425.61205.qm@web53702.mail.re2.yahoo.com> <827221.6704.qm@web53704.mail.re2.yahoo.com> <1285616691.5954.34.camel@localhost> <201009280157.17264.jacob@openend.se> <1285634613.5954.112.camel@localhost> <1285703348.5276.56.camel@localhost> <1285809701.5129.150.camel@localhost> Message-ID: Hi Paolo, On Fri, Oct 1, 2010 at 12:11 AM, Paolo Giarrusso wrote: > BTW, I am not used to bother authors by asking questions when reading > sources; but as far as I saw, it is considered "socially acceptable" > here, isn't it? Yes, absolutely :-) Indeed, asking by chat (#pypy on irc.freenode.net) is the best way. Please feel free to do it. Armin From anto.cuni at gmail.com Mon Oct 4 23:56:34 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 04 Oct 2010 23:56:34 +0200 Subject: [pypy-dev] [pypy-svn] r77589 - pypy/trunk/pypy/jit/metainterp In-Reply-To: <20101004213232.2F5A1282B90@codespeak.net> References: <20101004213232.2F5A1282B90@codespeak.net> Message-ID: <4CAA4D92.9000908@gmail.com> On 04/10/10 23:32, fijal at codespeak.net wrote: > def transform(op): > from pypy.jit.metainterp.history import AbstractDescr > - # Rename CALL_PURE to CALL. > + # Rename CALL_PURE and CALL_INVARIANT to CALL. > # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. > - if op.getopnum() == rop.CALL_PURE: > + if (op.getopnum() == rop.CALL_PURE or > + op.getopnum() == rop.CALL_LOOPINVARIANT): > op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, > op.getdescr()) > elif op.getopnum() == rop.VIRTUAL_REF: Hi, I just wanted to tell that since the resoperation-refactoring, there is now a nice "copy_and_change" method to be used exactly for the case where you want to replace an operation with a "similar" one. So, the code above should be written like: op = op.copy_and_change(rop.CALL, args=op.getarglist()[1:]) ciao, Anto From fijall at gmail.com Tue Oct 5 08:42:54 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 5 Oct 2010 08:42:54 +0200 Subject: [pypy-dev] [pypy-svn] r77589 - pypy/trunk/pypy/jit/metainterp In-Reply-To: <4CAA4D92.9000908@gmail.com> References: <20101004213232.2F5A1282B90@codespeak.net> <4CAA4D92.9000908@gmail.com> Message-ID: On Mon, Oct 4, 2010 at 11:56 PM, Antonio Cuni wrote: > On 04/10/10 23:32, fijal at codespeak.net wrote: > >> ? def transform(op): >> ? ? ? from pypy.jit.metainterp.history import AbstractDescr >> - ? ?# Rename CALL_PURE to CALL. >> + ? ?# Rename CALL_PURE and CALL_INVARIANT to CALL. >> ? ? ? # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. >> - ? ?if op.getopnum() == rop.CALL_PURE: >> + ? ?if (op.getopnum() == rop.CALL_PURE or >> + ? ? ? ?op.getopnum() == rop.CALL_LOOPINVARIANT): >> ? ? ? ? ? op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? op.getdescr()) >> ? ? ? elif op.getopnum() == rop.VIRTUAL_REF: > > Hi, > I just wanted to tell that since the resoperation-refactoring, there is now a > nice "copy_and_change" method to be used exactly for the case where you want > to replace an operation with a "similar" one. > > So, the code above should be written like: > > ? ?op = op.copy_and_change(rop.CALL, args=op.getarglist()[1:]) ah, great! :) > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From andrewfr_ice at yahoo.com Tue Oct 5 15:51:05 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Tue, 5 Oct 2010 06:51:05 -0700 (PDT) Subject: [pypy-dev] PyPy JIT & C extensions, greenlet Message-ID: <192780.17766.qm@web120710.mail.ne1.yahoo.com> Hi Armin and folks: --- On Thu, 9/30/10, Armin Rigo wrote: > From: Armin Rigo > Subject: Re: [pypy-dev] PyPy JIT & C extensions, greenlet > To: "Andrew Francis" > Cc: pypy-dev at codespeak.net > Date: Thursday, September 30, 2010, 5:25 AM > Maybe I should expand on an idea posted on #pypy by > fijal.? He mentioned that he would like to try to support Stackless in > PyPy without using the stackless transform, just by using the > same low-level stack hacks that are done by greenlet.c and > optionally by Stackless Python.? This means that there would be two > different approaches we can consider to support Stackless in PyPy: That sounds like a neat idea! > ? ? tasklet-switching Python code ???tasklet-switching Python code > ? ? becomes a single loop in? ? ?? ? becomes N loops with residual calls > ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? to machine code? ? ? ? Maybe this is the core of my problems (and ability to help) - I have a superficial understanding of how Stackless works under the hood. Where do the N loops come from? I have read Christian's "Stackless Python" paper (http://www.python.org/workshops/2000-01/proceedings/papers/tismers/stackless.htm) and slowly but surely I do more stuff with the C Stackless code base. That said, I haven't found any particularly good papers on how the Python interpreter itself is architected. Maybe I should look at how the greenlet package works. Again, I would be happy to help but I am not quite sure where to start so at least I can ask intelligent questions in the pypy IRC. Cheers, Andrew From brian at slesinsky.org Thu Oct 7 05:33:26 2010 From: brian at slesinsky.org (Brian Slesinsky) Date: Wed, 6 Oct 2010 20:33:26 -0700 Subject: [pypy-dev] ssl cert expired / OS X download doesn't work In-Reply-To: References: Message-ID: Hi, I tried running pypy 1.3 on OS X but ran into a few issues: First, I got this error: $ pypy dyld: Library not loaded: /usr/lib/libssl.0.9.8.dylib ?Referenced from: /usr/local/pypy-1.3/bin/pypy ?Reason: image not found Trace/BPT trap It looks like my machine (running OS X 10.5.8) has libssl.0.9.7.dylib, but not libssl.0.9.8. (I suppose I could recompile from source, but don't feel like waiting an hour.) Also, the SSL certificate has expired for https://codespeak.net, which is why I didn't post there. Anyway, please keep up the good work. - Brian From santagada at gmail.com Thu Oct 7 06:33:19 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 7 Oct 2010 01:33:19 -0300 Subject: [pypy-dev] ssl cert expired / OS X download doesn't work In-Reply-To: References: Message-ID: On Thu, Oct 7, 2010 at 12:33 AM, Brian Slesinsky wrote: > Hi, I tried running pypy 1.3 on OS X but ran into a few issues: > > First, I got this error: > > $ pypy > dyld: Library not loaded: /usr/lib/libssl.0.9.8.dylib > ?Referenced from: /usr/local/pypy-1.3/bin/pypy > ?Reason: image not found > Trace/BPT trap > > It looks like my machine (running OS X 10.5.8) has libssl.0.9.7.dylib, > but not libssl.0.9.8. (I suppose I could recompile from source, but > don't feel like waiting an hour.) The binaries are for mac os x 10.6 intel 32 bits. Someone should look at compiling it using the SDK (so we can choose 10.5 or 10.6 compatibility). > Also, the SSL certificate has expired for https://codespeak.net, which > is why I didn't post there. Yes it expired 17 of august. -- Leonardo Santagada From holger at merlinux.eu Sun Oct 10 09:37:49 2010 From: holger at merlinux.eu (holger krekel) Date: Sun, 10 Oct 2010 09:37:49 +0200 Subject: [pypy-dev] issue tracker spam issues / switching Message-ID: <20101010073749.GB11924@trillke.net> Hi all, seems like the pypy-dev issue tracker got some spam through bot-registered users. Unless i am mistaken there is no easy way to add a captcha in addition to the e-mail confirmation registration and so i made the default role for new issue tracker users "anonymous". This means new users need to find an admin who assigns the role "User". Admins on the trackers are e.g. Armin, Maciej, Anto, Amaury, myself and probably others. Not a great solution, i know. It doesn't affect any of the existing users, though. it's probably a good occassion to push for switching to using new issue tracker software especially since there has been other complaints as well. best, holger From fijall at gmail.com Mon Oct 11 11:42:20 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 11 Oct 2010 11:42:20 +0200 Subject: [pypy-dev] issue tracker spam issues / switching In-Reply-To: <20101010073749.GB11924@trillke.net> References: <20101010073749.GB11924@trillke.net> Message-ID: On Sun, Oct 10, 2010 at 9:37 AM, holger krekel wrote: > Hi all, > > seems like the pypy-dev issue tracker got some spam through bot-registered > users. ?Unless i am mistaken there is no easy way to add a captcha in addition > to the e-mail confirmation registration and so i made the default role for new > issue tracker users "anonymous". ?This means new users need to find an admin who > assigns the role "User". ?Admins on the trackers are e.g. Armin, Maciej, Anto, > Amaury, myself and probably others. ?Not a great solution, i know. It doesn't > affect any of the existing users, though. > > it's probably a good occassion to push for switching to using new > issue tracker software especially since there has been other complaints > as well. Hey. Yes, I think so, especially if we can give handling of spam to someone else ;-) I did a bit of research at some point and personally I would like to give JIRA a try. It looks good, but it's not open source (it's free for open source users though). Cheers, fijal > > best, > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Sun Oct 17 10:29:58 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 17 Oct 2010 10:29:58 +0200 Subject: [pypy-dev] [pypy-svn] r77933 - pypy/branch/jitffi/pypy/jit/metainterp/test In-Reply-To: <20101014133223.A97D9282BEA@codespeak.net> References: <20101014133223.A97D9282BEA@codespeak.net> Message-ID: The exception still can't happen though. On Thu, Oct 14, 2010 at 3:32 PM, wrote: > Author: antocuni > Date: Thu Oct 14 15:32:22 2010 > New Revision: 77933 > > Modified: > ? pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py > Log: > update this test too > > > Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py > ============================================================================== > --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py (original) > +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py Thu Oct 14 15:32:22 2010 > @@ -77,7 +77,9 @@ > ? ? ? ? call(0, p2, ? ? ? ? ? ? descr=libffi_prepare) > ? ? ? ? call(0, p2, i0, ? ? ? ? descr=libffi_push_arg) > ? ? ? ? call(0, p2, f1, ? ? ? ? descr=libffi_push_arg) > - ? ? ? ?i3 = call(0, p2, 12345, descr=libffi_call) > + ? ? ? ?i3 = call_may_force(0, p2, 12345, descr=libffi_call) > + ? ? ? ?guard_not_forced() [] > + ? ? ? ?guard_no_exception() [] > ? ? ? ? jump(i3, f1, p2) > ? ? ? ? """ > ? ? ? ? expected = ops > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From andrewfr_ice at yahoo.com Mon Oct 25 05:12:21 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Sun, 24 Oct 2010 20:12:21 -0700 (PDT) Subject: [pypy-dev] PyPy JIT & C extensions Message-ID: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Hi Folks: > --- On Thu, 9/30/10, Armin Rigo > wrote: > > > From: Armin Rigo > > Subject: Re: [pypy-dev] PyPy JIT & C extensions, > greenlet > > To: "Andrew Francis" > > Cc: pypy-dev at codespeak.net > > Date: Thursday, September 30, 2010, 5:25 AM > > > > Maybe I should expand on an idea posted on #pypy by > > fijal. He mentioned that he would like to try to > support Stackless in PyPy without using the stackless transform, >>just by using the same low-level stack hacks that are done by greenlet.c > and optionally by Stackless Python. This means that there would be two > > different approaches we can consider to support > Stackless in PyPy: A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, install the greenlet package, run it and see what happens? After all, greenlet is a C extension module. Hopefully this would provide a base line for work to be required? I am trying this but I am having a few hiccups getting setup.py to do the right thing. Since I am a newbie, I would like to know what is wrong with this line of reasoning? Cheers, Andrew From william.leslie.ttg at gmail.com Mon Oct 25 05:36:01 2010 From: william.leslie.ttg at gmail.com (William Leslie) Date: Mon, 25 Oct 2010 14:36:01 +1100 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: <365555.71903.qm@web120710.mail.ne1.yahoo.com> References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: On 25 October 2010 14:12, Andrew Francis wrote: > A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, install the greenlet package, run it and see what happens? After all, greenlet is a C extension module. ?Hopefully this would provide a base line for work to be required? I am trying this but I am having a few hiccups getting setup.py to do the right thing. Since I am a newbie, I would like to know what is wrong with this line of reasoning? PyPy's internal representation of frames differs from that of CPython. How does greenlet determine how much of the stack to copy? -- William Leslie From arigo at tunes.org Mon Oct 25 14:08:20 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Oct 2010 14:08:20 +0200 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: <365555.71903.qm@web120710.mail.ne1.yahoo.com> References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: Hi, On Mon, Oct 25, 2010 at 5:12 AM, Andrew Francis wrote: > A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, > install the greenlet package, run it and see what happens? You will just get strange segfaults and no way to understand from outside where they come from. I can give some more details about them, but I guess it would not be too helpful. In summary, C extension modules have a chance to work with PyPy as long as they don't play strange tricks. A bient?t, Armin. From fijall at gmail.com Mon Oct 25 16:37:28 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Oct 2010 16:37:28 +0200 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: On Mon, Oct 25, 2010 at 2:08 PM, Armin Rigo wrote: > Hi, > > On Mon, Oct 25, 2010 at 5:12 AM, Andrew Francis wrote: >> A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, >> install the greenlet package, run it and see what happens? > > You will just get strange segfaults and no way to understand from > outside where they come from. ?I can give some more details about > them, but I guess it would not be too helpful. ?In summary, C > extension modules have a chance to work with PyPy as long as they > don't play strange tricks. > Hey. I had a private email exchange with Andrew and I think what he wants to achieve is a working greenlet module for PyPy. Is a current C version a good start, or something else would be a good start? Cheers, fijal > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Mon Oct 25 16:50:50 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Oct 2010 16:50:50 +0200 Subject: [pypy-dev] [pypy-svn] r78267 - pypy/branch/arm-backend/pypy/jit/backend/arm/tools In-Reply-To: <20101025144559.82B2A282BE3@codespeak.net> References: <20101025144559.82B2A282BE3@codespeak.net> Message-ID: Hey. Can we make this directory 'tool' instead of 'tools' like every other tool directory? Cheers, fijal On Mon, Oct 25, 2010 at 4:45 PM, wrote: > Author: david > Date: Mon Oct 25 16:45:58 2010 > New Revision: 78267 > > Added: > ? pypy/branch/arm-backend/pypy/jit/backend/arm/tools/ > ? pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py ? (contents, props changed) > Log: > Wrapper for calling objdump on dumped memory > > Added: pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py > ============================================================================== > --- (empty file) > +++ pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py ? ? ? Mon Oct 25 16:45:58 2010 > @@ -0,0 +1,5 @@ > +#!/usr/bin/env python > +import os > +import sys > + > +os.system('objdump -D --architecture=arm --target=binary %s' % sys.argv[1]) > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From arigo at tunes.org Mon Oct 25 16:59:50 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Oct 2010 16:59:50 +0200 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: Hi, On Mon, Oct 25, 2010 at 4:37 PM, Maciej Fijalkowski wrote: > I had a private email exchange with Andrew and I think what he wants > to achieve is a working greenlet module for PyPy. Is a current C > version a good start, or something else would be a good start? Sorry, I guess I should repeat here the argument I made on IRC. It would not work, because the GC will not find its roots. With --gcrootfinder=shadowstack, I think it will get confused by the switches -- maybe I am wrong and it works anyway; but then I know for sure that with --gcrootfinder=asmgcc it will definitely not work at all. (For reference, the JIT requires --gcrootfinder=asmgcc.) Armin From andrewfr_ice at yahoo.com Mon Oct 25 19:36:26 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Mon, 25 Oct 2010 10:36:26 -0700 (PDT) Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: Message-ID: <733520.43651.qm@web120703.mail.ne1.yahoo.com> Hi Folks: --- On Mon, 10/25/10, Maciej Fijalkowski wrote: > From: Maciej Fijalkowski > Subject: Re: [pypy-dev] PyPy JIT & C extensions > To: "Armin Rigo" > Cc: "Andrew Francis" , pypy-dev at codespeak.net > Date: Monday, October 25, 2010, 7:37 AM > On Mon, Oct 25, 2010 at 2:08 PM, > Armin Rigo > wrote: > > Hi, > I had a private email exchange with Andrew and I think what > he wants to achieve is a working greenlet module for PyPy. Is a > current C version a good start, or something else would be a good > start? Yes this is exactly what I wish to achieve. I thought starting with the current greenlet module would be a good start. Maybe we will get lucky and not need to re-invent the wheel. I would learn the terminology (i.e., shadowstacks), see how differently the stacks are laid out, get familiar with the code base and debugging techniques. Already I see interesting questions being asked. Cheers, Andrew