From abrenon at wyplay.com Wed Jul 3 16:04:34 2013 From: abrenon at wyplay.com (Alexis BRENON) Date: Wed, 03 Jul 2013 16:04:34 +0200 Subject: [pypy-dev] Pypy translation fails on MIPS (without JIT) Message-ID: <51D42F72.3030501@wyplay.com> Hi everyone, As someones know, I'm working for a company (as an internship) on the MIPS port of Pypy. I started the JIT backend but not finished and for the moment I try to translate Pypy without JIT (because JIT isn't finished). I try to translate it directly on the target, even if it will take very very long time. But for the moment I only try to compile a small Rpython snippet : def main(args): print "Hello World" return 0 def target(*args): return main, None with this command line : python ../../rpython/bin/rpython -O1 test.py Then, I've got this translation configuration : [translation] translate.py configuration: [translation] [translate] [translation] opt = 1 [translation] targetspec = test [translation] translation configuration: [translation] [translation] [translation] [backendopt] [translation] inline_threshold = 16.2 [translation] continuation = False [translation] gc = boehm [translation] gcremovetypeptr = False [translation] gcrootfinder = n/a [translation] gctransformer = boehm [translation] list_comprehension_operations = True Annotating&Simplifying is OK, as RTyping, backend optimisations, stack checking, database creation and C generation. But during compilation, the first instruction fails... [platform:execute] make -j 2 in /tmp/usession-unknown-12/testing_1 [platform:Error] /opt/cross-native-mipsel-linux-gnu/bin/../lib/gcc/mipsel-linux-gnu/4.4.6/../../../../mipsel-linux-gnu/bin/ld: test-c: local symbol `__data_start' in /opt/cross-native-mipsel-linux-gnu/bin/../mipsel-linux-gnu/sysroot/usr/lib/crt1.o is referenced by DSO [platform:Error] /opt/cross-native-mipsel-linux-gnu/bin/../lib/gcc/mipsel-linux-gnu/4.4.6/../../../../mipsel-linux-gnu/bin/ld: final link failed: Bad value [platform:Error] collect2: ld returned 1 exit status [platform:Error] make: *** [test-c] Error 1 I search on Google and all I found is that it can comes from binutils 2.21, I try to create a new toolchain with binutils 2.22 but same error... Is anyone got any idea where I can search to debug this ? Thanks, Alexis From estama at gmail.com Wed Jul 3 17:47:45 2013 From: estama at gmail.com (Eleytherios Stamatogiannakis) Date: Wed, 03 Jul 2013 18:47:45 +0300 Subject: [pypy-dev] Open extremely slow when using "rU" flags In-Reply-To: <51D42F72.3030501@wyplay.com> References: <51D42F72.3030501@wyplay.com> Message-ID: <51D447A1.9030503@gmail.com> Hello to all, and thanks for the great work on PyPy. There remains a very problematic case in PyPy, where PyPy (tested with nightly) is around 193 times slower than CPython. This happens when reading a file using the universal newline flag ("rU"). Are there any plans to fix this problematic case, or should we just avoid using it when running under PyPy? Kind regards, lefteris. From estama at gmail.com Wed Jul 3 18:09:06 2013 From: estama at gmail.com (Eleytherios Stamatogiannakis) Date: Wed, 03 Jul 2013 19:09:06 +0300 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: <51D42F72.3030501@wyplay.com> References: <51D42F72.3030501@wyplay.com> Message-ID: <51D44CA2.1040803@gmail.com> Hello, We also found a case where PyPy is 2x slower than CPython. The following code: <<<< import cPickle fileIter=open("pypytesting", "w+b") mylist = ["qwerty"] * 100 for i in xrange(1000000): cPickle.dump(mylist, fileIter,1) >>>> Runs at: CPython 2.7.3: 13.114 sec PyPy nightly: 29.239 sec [Warning: it'll produce a file (pypytesting) that is 205 MB in size] Kind regards, lefteris. From amauryfa at gmail.com Wed Jul 3 18:16:44 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 3 Jul 2013 18:16:44 +0200 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: <51D44CA2.1040803@gmail.com> References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> Message-ID: 2013/7/3 Eleytherios Stamatogiannakis > Hello, > > We also found a case where PyPy is 2x slower than CPython. The following > code: > This is because of I/O. If I replace the file with a custom class which has an empty write() method, pypy is twice faster than CPython. Note: with pypy, io.open() is even slower :-( > <<<< > > import cPickle > > fileIter=open("pypytesting", "w+b") > mylist = ["qwerty"] * 100 > > for i in xrange(1000000): > cPickle.dump(mylist, fileIter,1) > > >>>> > > Runs at: > CPython 2.7.3: 13.114 sec > PyPy nightly: 29.239 sec > > [Warning: it'll produce a file (pypytesting) that is 205 MB in size] > > Kind regards, > > lefteris. > ______________________________**_________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/**mailman/listinfo/pypy-dev > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From estama at gmail.com Wed Jul 3 18:20:31 2013 From: estama at gmail.com (Eleytherios Stamatogiannakis) Date: Wed, 03 Jul 2013 19:20:31 +0300 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> Message-ID: <51D44F4F.5000500@gmail.com> On 03/07/13 19:16, Amaury Forgeot d'Arc wrote: > > 2013/7/3 Eleytherios Stamatogiannakis > > > Hello, > > We also found a case where PyPy is 2x slower than CPython. The > following code: > > > This is because of I/O. > If I replace the file with a custom class which has an empty write() method, > pypy is twice faster than CPython. Ah, i guess that the I/O isn't so much optimized, as the other things are, on PyPy? Thanks for looking into this test case. l. > Note: with pypy, io.open() is even slower :-( > > <<<< > > import cPickle > > fileIter=open("pypytesting", "w+b") > mylist = ["qwerty"] * 100 > > for i in xrange(1000000): > cPickle.dump(mylist, fileIter,1) > > >>>> > > Runs at: > CPython 2.7.3: 13.114 sec > PyPy nightly: 29.239 sec > > [Warning: it'll produce a file (pypytesting) that is 205 MB in size] > > Kind regards, > > lefteris. > _________________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/__mailman/listinfo/pypy-dev > > > > > > -- > Amaury Forgeot d'Arc From fijall at gmail.com Wed Jul 3 18:46:25 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 3 Jul 2013 18:46:25 +0200 Subject: [pypy-dev] Open extremely slow when using "rU" flags In-Reply-To: <51D447A1.9030503@gmail.com> References: <51D42F72.3030501@wyplay.com> <51D447A1.9030503@gmail.com> Message-ID: On Wed, Jul 3, 2013 at 5:47 PM, Eleytherios Stamatogiannakis wrote: > Hello to all, and thanks for the great work on PyPy. > > There remains a very problematic case in PyPy, where PyPy (tested with > nightly) is around 193 times slower than CPython. > > This happens when reading a file using the universal newline flag ("rU"). > > Are there any plans to fix this problematic case, or should we just avoid > using it when running under PyPy? > > Kind regards, > > lefteris. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev Hey. Well obviously first someone has to report the issue, which you just did. Can you please file a bug report with a standalone program that shows the behavior and put it on bugs.pypy.org? Thanks for your interest in pypy! Cheers, fijal From matti.picus at gmail.com Wed Jul 3 21:29:47 2013 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 03 Jul 2013 22:29:47 +0300 Subject: [pypy-dev] subclassing ndarray, was Memap in numpypy Message-ID: <51D47BAB.6040007@gmail.com> I got motivated and implemented parts of subtypes on ndarray in the ndarray-subtype branch. This page http://docs.scipy.org/doc/numpy/user/basics.subclassing.html describes the numpy way, so I did ndarray.view(subtype) as well, but I did not yet do the slice (they call it template) nor did I do __array_finalize__ Failing tests and patches are welcome Matti From anto.cuni at gmail.com Thu Jul 4 08:12:16 2013 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 4 Jul 2013 08:12:16 +0200 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> Message-ID: Il giorno 03/lug/2013 18:17, "Amaury Forgeot d'Arc" ha scritto: > This is because of I/O. > If I replace the file with a custom class which has an empty write() method, > pypy is twice faster than CPython. Few days ago I discovered that there is an easy optimization for this. If you look at how str2charp & friends are implemented, you see that we do an RPython loop and copy char by char. By contrast, things like string concatenation are implemented using memcpy and are much faster (like 3-4 times, iirc). Sorry if I don't give more precise pointer, but I'm on my mobile phone :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu Jul 4 08:44:01 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 4 Jul 2013 08:44:01 +0200 Subject: [pypy-dev] Pypy translation fails on MIPS (without JIT) In-Reply-To: <51D42F72.3030501@wyplay.com> References: <51D42F72.3030501@wyplay.com> Message-ID: Hi Alexis, On Wed, Jul 3, 2013 at 4:04 PM, Alexis BRENON wrote: > Is anyone got any idea where I can search to debug this ? Check first that compiling programs works at all. Then, still with a 5-lines example .c, try to add options to gcc one at a time until you reach a very similar command-line to the one in the generated Makefile. I bet it crashes at some point. A bient?t, Armin. From abrenon at wyplay.com Thu Jul 4 09:48:44 2013 From: abrenon at wyplay.com (Alexis BRENON) Date: Thu, 04 Jul 2013 09:48:44 +0200 Subject: [pypy-dev] Pypy translation fails on MIPS (without JIT) In-Reply-To: References: <51D42F72.3030501@wyplay.com> Message-ID: <51D528DC.2020506@wyplay.com> Le 04/07/2013 08:44, Armin Rigo a ?crit : > Hi Alexis, > > On Wed, Jul 3, 2013 at 4:04 PM, Alexis BRENON wrote: >> Is anyone got any idea where I can search to debug this ? > Check first that compiling programs works at all. Then, still with a > 5-lines example .c, try to add options to gcc one at a time until you > reach a very similar command-line to the one in the generated > Makefile. I bet it crashes at some point. > > > A bient?t, > > Armin. Thanks Armin for you advice and I find where it crashes. When linking, there is this option: --version-script=../dynamic-symbols-1 I read this file and remembered that I already see a similar one when I was searching on Google. First this file looks like this : { global: rpython_startup_code; get_errno; set_errno; local: *; }; Adding the failing symbol in global make it works. { global: rpython_startup_code; get_errno; set_errno; __data_start; local: *; }; Nevertheless, the resulting executable, which must display "Hello World" (as usual), segfault when I launch it... From GDB I get this, when running it with a breakpoint on main() (so it fails before enterring in the main) : Program received signal SIGSEGV, Segmentation fault. 0x77fc86a4 in dl_main (phdr=, phnum=, user_entry=, auxv=0x7fff6e44) at rtld.c:1652 1652 rtld.c: No such file or directory. in rtld.c (gdb) bt #0 0x77fc86a4 in dl_main (phdr=, phnum=, user_entry=, auxv=0x7fff6e44) at rtld.c:1652 #1 0x77fdd560 in _dl_sysdep_start (start_argptr=, dl_main=0x77fc7a84 ) at ../elf/dl-sysdep.c:244 #2 0x77fca5c4 in _dl_start_final (arg=0x7fff6e00, info=) at rtld.c:336 #3 0x77fca860 in _dl_start (arg=0x7fff6e00) at rtld.c:564 #4 0x77fc6894 in __start () from /lib/ld.so.1 Backtrace stopped: frame did not save the PC This bug is the same as if translate my simple Rpython file with the -O2 option. The translation success, but I get the exactly same error on the resulting executable... I encouter this error when I translate (with -O2 option) the targetpypystandalone.py file too, when it tries to execute platcheck_0 during translation... Maybe there is a deeper reason of all these failures. But I can't point out which or where... Any idea ? Thanks, Alexis From arigo at tunes.org Thu Jul 4 18:14:48 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 4 Jul 2013 18:14:48 +0200 Subject: [pypy-dev] Pypy translation fails on MIPS (without JIT) In-Reply-To: <51D528DC.2020506@wyplay.com> References: <51D42F72.3030501@wyplay.com> <51D528DC.2020506@wyplay.com> Message-ID: Hi Alexis, On Thu, Jul 4, 2013 at 9:48 AM, Alexis BRENON wrote: > Maybe there is a deeper reason of all these failures. But I can't point out > which or where... Any idea ? The same as I already said in my previous reply. Try from the other side: write a 5-lines example C program, try to compile, and then progressively make it "more like" what the translation produces, notably in terms of options from the Makefile. A bient?t, Armin. From arigo at tunes.org Thu Jul 4 18:23:03 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 4 Jul 2013 18:23:03 +0200 Subject: [pypy-dev] stmgc: about major collections Message-ID: Hi Remi, Just a note about major collections and the handling of h_original in stmgc: as you said in an XXX we don't need to *trace* the h_original object, but only keep it alive. But also, I thought some time ago about a trick to improve the overall performance of prebuilt objects, which is to copy the latest public copy *back* over the original prebuilt copy. This idea can be combined with the goal of keeping h_original: we would always copy the current copy back over the copy designated by h_original, and free all other copies (including the current copy). A bient?t, Armin. From mike at tradeworx.com Thu Jul 4 20:57:24 2013 From: mike at tradeworx.com (Mike Beller) Date: Thu, 4 Jul 2013 14:57:24 -0400 Subject: [pypy-dev] subclassing ndarray, was Memap in numpypy In-Reply-To: <51D47BAB.6040007@gmail.com> References: <51D47BAB.6040007@gmail.com> Message-ID: Matti That's great. I was stuck trying to figure out how to deal with the fact that if subtype was not-none, the from_shape (and friends) would also need access to 'space'. I see you addressed that with subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy codebase. I keep thinking that there are already very specific pypy-ish ways of doing things at interpreter level, and digging to try to learn those patterns, when perhaps things are a bit more in flux than I thought. On awesome side, I think now with your subclassing code, and Armin's modification to mmap.py to support more of the buffer interface, the memap code I've already written may just work. Will try that next. Mike On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus wrote: > I got motivated and implemented parts of subtypes on ndarray in the > ndarray-subtype branch. > This page > http://docs.scipy.org/doc/**numpy/user/basics.subclassing.**html > describes the numpy way, so I did > ndarray.view(subtype) > as well, but I did not yet do the slice (they call it template) nor did I > do __array_finalize__ > > Failing tests and patches are welcome > > Matti > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Thu Jul 4 21:27:40 2013 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 04 Jul 2013 22:27:40 +0300 Subject: [pypy-dev] subclassing ndarray, was Memap in numpypy In-Reply-To: References: <51D47BAB.6040007@gmail.com> Message-ID: <51D5CCAC.3020809@gmail.com> You correctly point out, in other words, that it was an ugly hack. It is on the way to being removed by adding a space argument to calls to from_space...(). Let us know how your tests progress. Matti On 07/04/2013 09:57 PM, Mike Beller wrote: > Matti > > That's great. I was stuck trying to figure out how to deal with the > fact that if subtype was not-none, the from_shape (and friends) would > also need access to 'space'. I see you addressed that with > subtype_and_space=(None,None). It's not a pattern I'd seen in the > pypy codebase. I keep thinking that there are already very specific > pypy-ish ways of doing things at interpreter level, and digging to try > to learn those patterns, when perhaps things are a bit more in flux > than I thought. > > On awesome side, I think now with your subclassing code, and Armin's > modification to mmap.py to support more of the buffer interface, the > memap code I've already written may just work. Will try that next. > > Mike > > > > On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus > wrote: > > I got motivated and implemented parts of subtypes on ndarray in > the ndarray-subtype branch. > This page > http://docs.scipy.org/doc/numpy/user/basics.subclassing.html > describes the numpy way, so I did > ndarray.view(subtype) > as well, but I did not yet do the slice (they call it template) > nor did I do __array_finalize__ > > Failing tests and patches are welcome > > Matti > > From amauryfa at gmail.com Thu Jul 4 21:30:42 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 4 Jul 2013 21:30:42 +0200 Subject: [pypy-dev] subclassing ndarray, was Memap in numpypy In-Reply-To: <51D5CCAC.3020809@gmail.com> References: <51D47BAB.6040007@gmail.com> <51D5CCAC.3020809@gmail.com> Message-ID: Hi Matti, I think you still have to implement the __array_finalize__ protocol. Otherwise methods like .transpose() won't work on a memmap object. 2013/7/4 Matti Picus > You correctly point out, in other words, that it was an ugly hack. > It is on the way to being removed by adding a space argument to calls to > from_space...(). > Let us know how your tests progress. > Matti > > On 07/04/2013 09:57 PM, Mike Beller wrote: > >> Matti >> >> That's great. I was stuck trying to figure out how to deal with the fact >> that if subtype was not-none, the from_shape (and friends) would also need >> access to 'space'. I see you addressed that with >> subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy >> codebase. I keep thinking that there are already very specific pypy-ish >> ways of doing things at interpreter level, and digging to try to learn >> those patterns, when perhaps things are a bit more in flux than I thought. >> >> On awesome side, I think now with your subclassing code, and Armin's >> modification to mmap.py to support more of the buffer interface, the memap >> code I've already written may just work. Will try that next. >> >> Mike >> >> >> >> On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus > matti.picus at gmail.com>**> wrote: >> >> I got motivated and implemented parts of subtypes on ndarray in >> the ndarray-subtype branch. >> This page >> http://docs.scipy.org/doc/**numpy/user/basics.subclassing.**html >> describes the numpy way, so I did >> ndarray.view(subtype) >> as well, but I did not yet do the slice (they call it template) >> nor did I do __array_finalize__ >> >> Failing tests and patches are welcome >> >> Matti >> >> >> > ______________________________**_________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/**mailman/listinfo/pypy-dev > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Thu Jul 4 21:34:51 2013 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 04 Jul 2013 22:34:51 +0300 Subject: [pypy-dev] subclassing ndarray, was Memap in numpypy In-Reply-To: References: <51D47BAB.6040007@gmail.com> <51D5CCAC.3020809@gmail.com> Message-ID: <51D5CE5B.1040201@gmail.com> True. changeset fd71a983d387 includes the start of tests for __array_finalize__ and a "few" failing tests Help is welcome :) Matti On 07/04/2013 10:30 PM, Amaury Forgeot d'Arc wrote: > Hi Matti, > > I think you still have to implement the __array_finalize__ protocol. > Otherwise methods like .transpose() won't work on a memmap object. > > 2013/7/4 Matti Picus > > > You correctly point out, in other words, that it was an ugly hack. > It is on the way to being removed by adding a space argument to > calls to from_space...(). > Let us know how your tests progress. > Matti > > On 07/04/2013 09:57 PM, Mike Beller wrote: > > Matti > > That's great. I was stuck trying to figure out how to deal > with the fact that if subtype was not-none, the from_shape > (and friends) would also need access to 'space'. I see you > addressed that with subtype_and_space=(None,None). It's not > a pattern I'd seen in the pypy codebase. I keep thinking that > there are already very specific pypy-ish ways of doing things > at interpreter level, and digging to try to learn those > patterns, when perhaps things are a bit more in flux than I > thought. > > On awesome side, I think now with your subclassing code, and > Armin's modification to mmap.py to support more of the buffer > interface, the memap code I've already written may just work. > Will try that next. > > Mike > > > > On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus > > >> > wrote: > > I got motivated and implemented parts of subtypes on > ndarray in > the ndarray-subtype branch. > This page > http://docs.scipy.org/doc/numpy/user/basics.subclassing.html > describes the numpy way, so I did > ndarray.view(subtype) > as well, but I did not yet do the slice (they call it > template) > nor did I do __array_finalize__ > > Failing tests and patches are welcome > > Matti > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > > > > -- > Amaury Forgeot d'Arc From mike at tradeworx.com Thu Jul 4 23:11:06 2013 From: mike at tradeworx.com (Mike Beller) Date: Thu, 4 Jul 2013 17:11:06 -0400 Subject: [pypy-dev] subclassing ndarray, was Memap in numpypy In-Reply-To: <51D5CCAC.3020809@gmail.com> References: <51D47BAB.6040007@gmail.com> <51D5CCAC.3020809@gmail.com> Message-ID: Matti/Amaury Well -- that's excellent. The basic functionality of memmap.py seems to work fine now, against the ndarray-subtype branch. I had copied the tests from the original modules doctest into a pytest module. Had to modify because numpypy doesn't support .flags attribute, and also because offset is not supported yet. I am adding some additional tests to see what happens when sizes of things are wrong, and when one tries to write to read-only mmaps -- don't want to allow segfaults presumably. Other than that -- how do I go about making my code available for review / merging? Should I wait until Matti's branch is merged, then make a new branch called numpypy-memmap? Or should memmap just be part of the ndarray-subtype branch (since it is a user of the subtype functionality)? Mike On Thu, Jul 4, 2013 at 3:27 PM, Matti Picus wrote: > You correctly point out, in other words, that it was an ugly hack. > It is on the way to being removed by adding a space argument to calls to > from_space...(). > Let us know how your tests progress. > Matti > > On 07/04/2013 09:57 PM, Mike Beller wrote: > >> Matti >> >> That's great. I was stuck trying to figure out how to deal with the fact >> that if subtype was not-none, the from_shape (and friends) would also need >> access to 'space'. I see you addressed that with >> subtype_and_space=(None,None). It's not a pattern I'd seen in the pypy >> codebase. I keep thinking that there are already very specific pypy-ish >> ways of doing things at interpreter level, and digging to try to learn >> those patterns, when perhaps things are a bit more in flux than I thought. >> >> On awesome side, I think now with your subclassing code, and Armin's >> modification to mmap.py to support more of the buffer interface, the memap >> code I've already written may just work. Will try that next. >> >> Mike >> >> >> >> On Wed, Jul 3, 2013 at 3:29 PM, Matti Picus > matti.picus at gmail.com>**> wrote: >> >> I got motivated and implemented parts of subtypes on ndarray in >> the ndarray-subtype branch. >> This page >> http://docs.scipy.org/doc/**numpy/user/basics.subclassing.**html >> describes the numpy way, so I did >> ndarray.view(subtype) >> as well, but I did not yet do the slice (they call it template) >> nor did I do __array_finalize__ >> >> Failing tests and patches are welcome >> >> Matti >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abrenon at wyplay.com Fri Jul 5 10:00:16 2013 From: abrenon at wyplay.com (Alexis BRENON) Date: Fri, 05 Jul 2013 10:00:16 +0200 Subject: [pypy-dev] Pypy translation fails on MIPS (without JIT) In-Reply-To: References: <51D42F72.3030501@wyplay.com> <51D528DC.2020506@wyplay.com> Message-ID: <51D67D10.4030202@wyplay.com> Le 04/07/2013 18:14, Armin Rigo a ?crit : > Hi Alexis, > > On Thu, Jul 4, 2013 at 9:48 AM, Alexis BRENON wrote: >> Maybe there is a deeper reason of all these failures. But I can't point out >> which or where... Any idea ? > The same as I already said in my previous reply. Try from the other > side: write a 5-lines example C program, try to compile, and then > progressively make it "more like" what the translation produces, > notably in terms of options from the Makefile. > > > A bient?t, > > Armin. Hi, Well, I did some tests on a very simple C file that just prints "Hello World !". I did many option combinations and I could notice that : - with -Wl,--version-script=dynamic-symbols-1 LDFLAGS alone (with no other option), compilation works but execution segfault - this LDFLAGS seems to be OK with many other options (same result as previous) - as soon as I add -lgc option to LIBS, compilation fails with the __data_start error You can find tests and result on this pastebin : http://pastebin.com/F8NvC0Ex Thanks for your help. Alexis. From seanfisk at gmail.com Fri Jul 5 20:30:34 2013 From: seanfisk at gmail.com (Sean Fisk) Date: Fri, 5 Jul 2013 12:30:34 -0600 Subject: [pypy-dev] Threading in RPython Message-ID: Hello everyone, For my internship, I am working on implementing a solver for partial differential equations in RPython . I am investigating the possibility of parallelizing the code using multi-threading. I have found the RPython's basic threading library, rpython/rlib/rthread.py, and am attempting to get a "Hello world!" threading program up and running. I have successfully been able to start threads that do really simple things like printing literal messages. However, I would now like to be able to pass parameters to the threads. The only example I can seem to find of this library in use is the threading support in PyPy (at pypy/module/thread/os_thread.py). I?ve read through the code a number of times and am using it as a reference. However, there are some features I don?t need and some things that simply won't run in my interpreter (e.g., everything involving spaces). Can anybody point me in the right direction as to passing parameters to threads? I know I need something similar to the Bootstrapper to synchronize my parameters, but everything I've tried so far has either not synchronized or segfaulted. The current code is segfaulting around rthread.gc_thread_die(). I can do some more digging on this if necessary. Thanks in advance for help anyone might offer. ~ Sean $ python --version Python 2.7.3 (5acfe049a5b0, May 21 2013, 13:47:22) [PyPy 2.0.2 with GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] Here is the current code: # rthread_hello.py## Compile with: rpython --thread rthread_hello.py from rpython.rlib import rthread # "Library" code class Arguments(object): pass class FunctionData(object): args = Arguments() func = None lock = None _function_data = FunctionData() def _thread_func_wrapper(): rthread.gc_thread_start() func = _function_data.func args = _function_data.args _function_data.lock.release() func(args) rthread.gc_thread_die() def run_in_thread(func, args): _function_data.func = func _function_data.args = args if _function_data.lock is None: _function_data.lock = rthread.allocate_lock() _function_data.lock.acquire(True) rthread.gc_thread_prepare() rthread.start_new_thread(_thread_func_wrapper, ()) # "User" code def func_to_call(args): print 'Within the function' print args.first print args.second def main(argv): for i in xrange(5): args = Arguments() # Create some dummy arguments. args.first = i args.second = i * 10 # Start the thread. run_in_thread(func_to_call, args) return 0 def target(x, y): return main, None -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri Jul 5 23:17:15 2013 From: arigo at tunes.org (Armin Rigo) Date: Fri, 5 Jul 2013 23:17:15 +0200 Subject: [pypy-dev] Threading in RPython In-Reply-To: References: Message-ID: Hi Sean, On Fri, Jul 5, 2013 at 8:30 PM, Sean Fisk wrote: > For my internship, I am working on implementing a solver for partial > differential equations in RPython. I am investigating the possibility of > parallelizing the code using multi-threading. Sorry, RPython is the wrong tool. It doesn't support multithreading any more than CPython and PyPy: it relies on the GIL to avoid crashing, including (but not limited to) everything related to the GC. The best you can do is dividing the problem among multiple processes. A bient?t, Armin. From njh at njhurst.com Sat Jul 6 00:28:39 2013 From: njh at njhurst.com (Nathan Hurst) Date: Sat, 6 Jul 2013 08:28:39 +1000 Subject: [pypy-dev] Threading in RPython In-Reply-To: References: Message-ID: <20130705222839.GA25329@ajhurst.org> On Fri, Jul 05, 2013 at 11:17:15PM +0200, Armin Rigo wrote: > Hi Sean, > > On Fri, Jul 5, 2013 at 8:30 PM, Sean Fisk wrote: > > For my internship, I am working on implementing a solver for partial > > differential equations in RPython. I am investigating the possibility of > > parallelizing the code using multi-threading. > > Sorry, RPython is the wrong tool. It doesn't support multithreading > any more than CPython and PyPy: it relies on the GIL to avoid > crashing, including (but not limited to) everything related to the GC. > The best you can do is dividing the problem among multiple processes. Hi Sean, Your internship sounds very exciting! RPython is frowned upon for general coding (I believe because it is not a stable supported language target, it requires great care for managing the various resources, and provides little benefit over targetting pypy). I would modestly suggest that instead you should implement your solver on pypy and use your new found rpython skills to optimise the part of the code which is performing poorly (or at least feed back concise benchmarks into pypy). I think that the work on numpypy would have a considerable overlap with your needs. Ping me or the list if you need help, also tell us more about your project and its goals! njh From ronan.lamy at gmail.com Sun Jul 7 01:39:57 2013 From: ronan.lamy at gmail.com (Ronan Lamy) Date: Sun, 7 Jul 2013 01:39:57 +0200 Subject: [pypy-dev] Killing OOType? (was Re: Translating pypy on FreeBSD with CLI backend) In-Reply-To: References: <1507130.Uoj2OIuUtg@dragon.dg> <1725112.nlyumnnUK3@dragon.dg> <518AC2E9.6010200@gmx.de> <518B57C5.10001@gmail.com> Message-ID: 2013/5/9 Armin Rigo > Hi all, > > On Thu, May 9, 2013 at 10:01 AM, Antonio Cuni wrote: > > Although I have an emotional feeling with that piece of code, I think > that > > Alex is right. > > I also tend to agree. Killing stuff that nobody seriously cares about > is sad but good, particularly when it adds some otherwise-unnecessary > levels of abstractions everywhere. We should ideally wait e.g. one > month for feedback from other developers that may still have plans > there. > It's been almost 2 months. Can we kill it *now*? I've just started a branch: https://bitbucket.org/pypy/pypy/commits/branch/kill-ootype It looks like killing it properly requires some work. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Sun Jul 7 01:42:27 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 7 Jul 2013 09:42:27 +1000 Subject: [pypy-dev] Killing OOType? (was Re: Translating pypy on FreeBSD with CLI backend) In-Reply-To: References: <1507130.Uoj2OIuUtg@dragon.dg> <1725112.nlyumnnUK3@dragon.dg> <518AC2E9.6010200@gmx.de> <518B57C5.10001@gmail.com> Message-ID: Yes, I think we can kill it (if you want some help deleting code, let us know :P) Alex On Sun, Jul 7, 2013 at 9:39 AM, Ronan Lamy wrote: > 2013/5/9 Armin Rigo > >> Hi all, >> >> On Thu, May 9, 2013 at 10:01 AM, Antonio Cuni >> wrote: >> > Although I have an emotional feeling with that piece of code, I think >> that >> > Alex is right. >> >> I also tend to agree. Killing stuff that nobody seriously cares about >> is sad but good, particularly when it adds some otherwise-unnecessary >> levels of abstractions everywhere. We should ideally wait e.g. one >> month for feedback from other developers that may still have plans >> there. >> > > > It's been almost 2 months. Can we kill it *now*? > > I've just started a branch: > https://bitbucket.org/pypy/pypy/commits/branch/kill-ootype > It looks like killing it properly requires some work. > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sun Jul 7 09:31:16 2013 From: arigo at tunes.org (Armin Rigo) Date: Sun, 7 Jul 2013 09:31:16 +0200 Subject: [pypy-dev] Pypy translation fails on MIPS (without JIT) In-Reply-To: <51D67D10.4030202@wyplay.com> References: <51D42F72.3030501@wyplay.com> <51D528DC.2020506@wyplay.com> <51D67D10.4030202@wyplay.com> Message-ID: Hi Alexis, On Fri, Jul 5, 2013 at 10:00 AM, Alexis BRENON wrote: > Well, I did some tests on a very simple C file that just prints "Hello World > !". I did many option combinations and I could notice that : > - with -Wl,--version-script=dynamic-symbols-1 LDFLAGS alone (with no > other option), compilation works but execution segfault I cannot help here. You need someone that knows about gcc and ld on MIPS. If you can't find any help, try asking on stackoverflow.com. > - as soon as I add -lgc option to LIBS, compilation fails with the > __data_start error Same, but this is specifically about Boehm's libgc on MIPS. A bient?t, Armin. From arigo at tunes.org Sun Jul 7 09:42:22 2013 From: arigo at tunes.org (Armin Rigo) Date: Sun, 7 Jul 2013 09:42:22 +0200 Subject: [pypy-dev] Threading in RPython In-Reply-To: References: Message-ID: Hi Sean, On Fri, Jul 5, 2013 at 11:17 PM, Armin Rigo wrote: > Sorry, RPython is the wrong tool. It doesn't support multithreading > (...) I've updated the FAQ entry here to try to explain a bit more precisely the reason and motivations for which we, the "core PyPy group" of people, keep answering the question "How do I use RPython to..." with "Don't!". http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython A bient?t, Armin. From seanfisk at gmail.com Sun Jul 7 22:23:23 2013 From: seanfisk at gmail.com (Sean Fisk) Date: Sun, 7 Jul 2013 14:23:23 -0600 Subject: [pypy-dev] Threading in RPython In-Reply-To: References: Message-ID: Nathan and Armin: Thank you both for the advice! Sorry, I didn't make it clear: my project is actually an interpreter for a stencil-based language that is used for solving partial differential equations. So I think that RPython is the right tool to an extent. It worked very well for writing the interpreter (I also used Alex Gaynor's rply ). However, now I would like to parallelize the solver (i.e., the matrix operations). I sounds as if this is where RPython stops being the right tool for the job, since threading is useful only for concurrency and not speed, as in Python. Thank you for letting me know before I spent too much time on it! I suppose I should have guessed from the code I was writing. I am seeing two options, on which I welcome comment: - Continue using RPython and use rffito make calls to C, where I implement the solver function for high speed, optionally with threading or message-passing. I've used pthreads, OpenMP, and MPI a number of times before, but combining that with rffi might be a little over my head. - Switch to using PyPy with multiprocessing and/or numpypy. Happily, my code runs under PyPy and CPython albeit with some RPython libraries and some RPython-specific workarounds (e.g., boxes ). I will talk to my advisor, and other ideas are certainly welcome! Thank you, Sean On Sun, Jul 7, 2013 at 1:42 AM, Armin Rigo wrote: Hi Sean, > > On Fri, Jul 5, 2013 at 11:17 PM, Armin Rigo wrote: > > Sorry, RPython is the wrong tool. It doesn't support multithreading > > (...) > > I've updated the FAQ entry here to try to explain a bit more precisely > the reason and motivations for which we, the "core PyPy group" of > people, keep answering the question "How do I use RPython to..." with > "Don't!". > > > http://doc.pypy.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython > > > A bient?t, > > Armin. > -- Sean Fisk -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Tue Jul 9 00:41:44 2013 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 09 Jul 2013 00:41:44 +0200 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> Message-ID: <51DB4028.5020507@gmail.com> Hello Eleytherios, On 07/04/2013 08:12 AM, Antonio Cuni wrote: > > Il giorno 03/lug/2013 18:17, "Amaury Forgeot d'Arc" > ha scritto: > > > This is because of I/O. > > If I replace the file with a custom class which has an empty write() method, > > pypy is twice faster than CPython. > > Few days ago I discovered that there is an easy optimization for this. If you > look at how str2charp & friends are implemented, you see that we do an RPython > loop and copy char by char. > By contrast, things like string concatenation are implemented using memcpy and > are much faster (like 3-4 times, iirc). > Sorry if I don't give more precise pointer, but I'm on my mobile phone :-) could you try to rerun your benchmark on the improve-str2charp branch please? The benchmarks on speed.pypy.org shows some important speedup in e.g. twisted_tcp or raytrace_simple, which seems to contain a lot of write I/O, so it might help your case as well: http://speed.pypy.org/comparison/?exe=1%2BL%2Bdefault%2C1%2BL%2Bimprove-str2charp&ben=1%2C34%2C27%2C2%2C25%2C3%2C46%2C4%2C5%2C41%2C42%2C22%2C44%2C6%2C39%2C7%2C8%2C45%2C23%2C24%2C9%2C10%2C47%2C48%2C49%2C50%2C51%2C11%2C12%2C13%2C40%2C14%2C15%2C35%2C36%2C37%2C38%2C16%2C52%2C54%2C55%2C53%2C56%2C28%2C30%2C32%2C29%2C33%2C17%2C18%2C19%2C20%2C43&env=1&hor=true&bas=1%2BL%2Bdefault&chart=normal+bars ciao, Anto From estama at gmail.com Tue Jul 9 17:29:01 2013 From: estama at gmail.com (Eleytherios Stamatogiannakis) Date: Tue, 09 Jul 2013 18:29:01 +0300 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: <51DB4028.5020507@gmail.com> References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> <51DB4028.5020507@gmail.com> Message-ID: <51DC2C3D.3010004@gmail.com> On 09/07/13 01:41, Antonio Cuni wrote: > Hello Eleytherios, > > On 07/04/2013 08:12 AM, Antonio Cuni wrote: >> >> Il giorno 03/lug/2013 18:17, "Amaury Forgeot d'Arc" > > ha scritto: >> >> > This is because of I/O. >> > If I replace the file with a custom class which has an empty >> write() method, >> > pypy is twice faster than CPython. >> >> Few days ago I discovered that there is an easy optimization for this. >> If you >> look at how str2charp & friends are implemented, you see that we do an >> RPython >> loop and copy char by char. >> By contrast, things like string concatenation are implemented using >> memcpy and >> are much faster (like 3-4 times, iirc). >> Sorry if I don't give more precise pointer, but I'm on my mobile phone >> :-) > > could you try to rerun your benchmark on the improve-str2charp branch > please? > The benchmarks on speed.pypy.org shows some important speedup in e.g. > twisted_tcp or raytrace_simple, which seems to contain a lot of write > I/O, so it might help your case as well: > > http://speed.pypy.org/comparison/?exe=1%2BL%2Bdefault%2C1%2BL%2Bimprove-str2charp&ben=1%2C34%2C27%2C2%2C25%2C3%2C46%2C4%2C5%2C41%2C42%2C22%2C44%2C6%2C39%2C7%2C8%2C45%2C23%2C24%2C9%2C10%2C47%2C48%2C49%2C50%2C51%2C11%2C12%2C13%2C40%2C14%2C15%2C35%2C36%2C37%2C38%2C16%2C52%2C54%2C55%2C53%2C56%2C28%2C30%2C32%2C29%2C33%2C17%2C18%2C19%2C20%2C43&env=1&hor=true&bas=1%2BL%2Bdefault&chart=normal+bars The times that we got with improve-str2charp, are a little worse than the previous nightly build that i had tried with in my previous email. I have rerun the benchmark and the times (best of 3 runs) are: CPython 2.7.3: 14.173 sec PyPy nightly 3/7/2013: 32.105 sec PyPy improve-str2charp: 34.044 sec Regards, l. From fijall at gmail.com Thu Jul 11 14:30:34 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 11 Jul 2013 14:30:34 +0200 Subject: [pypy-dev] PyPy 2.1 beta released Message-ID: =============== PyPy 2.1 beta 1 =============== We're pleased to announce the first beta of the upcoming 2.1 release of PyPy. This beta contains many bugfixes and improvements, numerous improvements to the numpy in PyPy effort. The main feature being that the ARM processor support is not longer considered alpha level. We would like to thank the Raspberry PiFoundation (http://www.raspberrypi.org) for supporting the work to finish PyPy's ARM support. You can download the PyPy 2.1 beta 1 release here: http://pypy.org/download.html Highlights ========== * Bugfixes to the ARM JIT backend, so that ARM is now an officially supported processor architecture * Stacklet support on ARM * Interpreter improvements * Various numpy improvements * Bugfixes to cffi and ctypes * Bugfixes to the stacklet support * Improved logging performance * Faster sets for objects What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3. It's fast due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Also this release supports ARM machines running Linux 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` that supports ``VFPv3`` should work. Cheers, the PyPy team From david.schneider at bivab.de Thu Jul 11 13:46:17 2013 From: david.schneider at bivab.de (David Schneider) Date: Thu, 11 Jul 2013 13:46:17 +0200 Subject: [pypy-dev] PyPy 2.1 beta 1 released Message-ID: <97C9FC70-B435-44DE-BC92-28DB9BC4DD0D@bivab.de> =============== PyPy 2.1 beta 1 =============== We're pleased to announce the first beta of the upcoming 2.1 release of PyPy. This beta contains many bugfixes and improvements, numerous improvements to the numpy in PyPy effort. The main feature being that the ARM processor support is not longer considered alpha level. We would like to thank the Raspberry PiFoundation (http://www.raspberrypi.org) for supporting the work to finish PyPy's ARM support. You can download the PyPy 2.1 beta 1 release here: http://pypy.org/download.html Highlights ========== * Bugfixes to the ARM JIT backend, so that ARM is now an officially supported processor architecture * Stacklet support on ARM * Interpreter improvements * Various numpy improvements * Bugfixes to cffi and ctypes * Bugfixes to the stacklet support * Improved logging performance * Faster sets for objects What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3. It's fast due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Also this release supports ARM machines running Linux 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` that supports ``VFPv3`` should work. Cheers, the PyPy team From matti.picus at gmail.com Tue Jul 16 19:02:38 2013 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 16 Jul 2013 20:02:38 +0300 Subject: [pypy-dev] ndarray-subtype branch looking for review Message-ID: <51E57CAE.50501@gmail.com> The ndarray-branch passes tests both translated and untranslated. I ran it through the numpy test suite as well, specifically the matrix class tests, and it seems to pass them. Would anyone like to review, or should I just go ahead and merge? Specifically, I may have missed some w_ prefixes on app-level instances, or may not have tested all the code paths, and I had to hack the FakeSpace in compile.py to allow subclassing. Matti From amauryfa at gmail.com Tue Jul 16 19:17:22 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 16 Jul 2013 19:17:22 +0200 Subject: [pypy-dev] ndarray-subtype branch looking for review In-Reply-To: <51E57CAE.50501@gmail.com> References: <51E57CAE.50501@gmail.com> Message-ID: I could do it. Is there a way to get a full diff of the changes about to be merged? 2013/7/16 Matti Picus > The ndarray-branch passes tests both translated and untranslated. I ran it > through the numpy test suite as well, specifically the matrix class tests, > and it seems to pass them. > Would anyone like to review, or should I just go ahead and merge? > Specifically, I may have missed some w_ prefixes on app-level instances, > or may not have tested all the code paths, and I had to hack the FakeSpace > in compile.py to allow subclassing. > Matti > ______________________________**_________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/**mailman/listinfo/pypy-dev > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Tue Jul 16 19:21:17 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 16 Jul 2013 19:21:17 +0200 Subject: [pypy-dev] ndarray-subtype branch looking for review In-Reply-To: References: <51E57CAE.50501@gmail.com> Message-ID: 2013/7/16 Amaury Forgeot d'Arc > I could do it. > Is there a way to get a full diff of the changes about to be merged? > Ok, found it. https://bitbucket.org/pypy/pypy/compare/ndarray-subtype..default#diff -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jul 17 01:44:17 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 17 Jul 2013 01:44:17 +0200 Subject: [pypy-dev] ndarray-subtype branch looking for review In-Reply-To: References: <51E57CAE.50501@gmail.com> Message-ID: you go to your checkout (default), you do hg merge and then hg diff. if you don't like it, you do hg up -C to clean stuff up. Easy to do in a separate dir (I suggest hg share for that) On Tue, Jul 16, 2013 at 7:17 PM, Amaury Forgeot d'Arc wrote: > I could do it. > Is there a way to get a full diff of the changes about to be merged? > > > 2013/7/16 Matti Picus >> >> The ndarray-branch passes tests both translated and untranslated. I ran it >> through the numpy test suite as well, specifically the matrix class tests, >> and it seems to pass them. >> Would anyone like to review, or should I just go ahead and merge? >> Specifically, I may have missed some w_ prefixes on app-level instances, >> or may not have tested all the code paths, and I had to hack the FakeSpace >> in compile.py to allow subclassing. >> Matti >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > > > > -- > Amaury Forgeot d'Arc > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From amauryfa at gmail.com Wed Jul 17 17:36:54 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 17 Jul 2013 17:36:54 +0200 Subject: [pypy-dev] ndarray-subtype branch looking for review In-Reply-To: References: <51E57CAE.50501@gmail.com> Message-ID: 2013/7/17 Maciej Fijalkowski > you go to your checkout (default), you do hg merge and > then hg diff. if you don't like it, you do hg up -C to clean stuff up. > Easy to do in a separate dir (I suggest hg share for that) > Actually I created a Pull Request: https://bitbucket.org/pypy/pypy/pull-request/164/ndarray-subtype-branch-review/diff Easy to comment on each diff line. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Wed Jul 17 19:02:29 2013 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 17 Jul 2013 20:02:29 +0300 Subject: [pypy-dev] ndarray-subtype branch looking for review In-Reply-To: References: <51E57CAE.50501@gmail.com> Message-ID: <51E6CE25.2010008@gmail.com> On 07/17/2013 06:36 PM, Amaury Forgeot d'Arc wrote: > 2013/7/17 Maciej Fijalkowski > > > you go to your checkout (default), you do hg merge and > then hg diff. if you don't like it, you do hg up -C to clean stuff up. > Easy to do in a separate dir (I suggest hg share for that) > > > Actually I created a Pull Request: > https://bitbucket.org/pypy/pypy/pull-request/164/ndarray-subtype-branch-review/diff > Easy to comment on each diff line. > > -- > Amaury Forgeot d'Arc Thanks for the review. How do I commit changes? Can I just commit to the branch, or should I do something on the pull request? Matti From cfbolz at gmx.de Thu Jul 18 11:08:56 2013 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 18 Jul 2013 11:08:56 +0200 Subject: [pypy-dev] PyPy doesn't make code written in C faster In-Reply-To: <20130629180625.GB4844@ajhurst.org> References: <20130530164102.GA13899@ajhurst.org> <51CA92D9.8040007@gmx.de> <20130629141907.GA4844@ajhurst.org> <51CEEEB6.8040803@gmx.de> <20130629180625.GB4844@ajhurst.org> Message-ID: <51E7B0A8.2050903@gmx.de> On 29/06/13 20:06, Nathan Hurst wrote: > I don't actually see any difference between that code and normal > python. What makes it rpython? Being RPython is mostly a contextual property. It does nothing "too dynamic", so it is RPython. >> The main changes I did from your version was generalize it to work for >> any 3 <= n <= 36, optimize the handling of leading zeros and use a >> StringBuilder instead of a list to build the resulting string. Plus I >> kept the already existing fast version for powers of two alive. > > All good decisions. Is there a nice way to profile this with > something like line_profiler? It's possible to improve the > asymptotics in various places, but they almost certainly aren't worth > it. For rbigint I just profiled the generated C source. > Of course the real question is whether pypy now beats python2.7 on the > all important factorial benchmark. (And if so should we rush out a > new release? ;) Yes, it does, and it will be in the 2.1 release. I also did some additional changes yesterday: https://bitbucket.org/pypy/pypy/commits/7ceb58dbdc244439b58d8bcc2d5e41a35fa3897b (and following commits) I now cache the powers of base needed to split the number, because the profile was completely dominated by calls to pow. In addition, I compute the minimum digits based on the base, instead of using a pessimistic bound for all bases. Now the profile is actually dominated by calls to divmod, which is probably expected. Cheers, Carl Friedrich From estama at gmail.com Thu Jul 18 12:38:53 2013 From: estama at gmail.com (Eleytherios Stamatogiannakis) Date: Thu, 18 Jul 2013 13:38:53 +0300 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> Message-ID: <51E7C5BD.3080308@gmail.com> We just found something strange that is going on with the test that we had posted. Due to Amaury's suspicion we started looking into PyPy's I/O speed. It turns out it is more or less the same speed as Python's: <<<< f=open("pypytesting", "w+b") mylist = str(["qwerty"] * 100) for i in xrange(1000000): f.write(mylist) >>>> Running at: CPython 2.7.3: 12.563 sec PyPy nightly: 12.492 total sec The previous code that we had posted (you can see it in previous email) does a: "" for i in xrange(1000000): cPickle.dump(mylist, f,1) "" And runs at: CPython 2.7.3: 13.114 sec PyPy nightly: 29.239 sec If we change previous code to write in another equivalent way: "" for i in xrange(1000000): f.write(cPickle.dumps(mylist,1)) "" Then the times are the same between CPython and PyPy: Cpython 2.7.3: 12.802 sec PyPy nightly: 12.181 sec Why is there such a huge speed difference between cPickle.dump( ... f) and f.write(cPickle.dumps(...)) ? Kind regards. l. On 03/07/13 19:16, Amaury Forgeot d'Arc wrote: > > 2013/7/3 Eleytherios Stamatogiannakis > > > Hello, > > We also found a case where PyPy is 2x slower than CPython. The > following code: > > > This is because of I/O. > If I replace the file with a custom class which has an empty write() method, > pypy is twice faster than CPython. > > Note: with pypy, io.open() is even slower :-( > > <<<< > > import cPickle > > fileIter=open("pypytesting", "w+b") > mylist = ["qwerty"] * 100 > > for i in xrange(1000000): > cPickle.dump(mylist, fileIter,1) > > >>>> > > Runs at: > CPython 2.7.3: 13.114 sec > PyPy nightly: 29.239 sec > > [Warning: it'll produce a file (pypytesting) that is 205 MB in size] > > Kind regards, > > lefteris. > _________________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/__mailman/listinfo/pypy-dev > > > > > > -- > Amaury Forgeot d'Arc From amauryfa at gmail.com Thu Jul 18 12:53:25 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 18 Jul 2013 12:53:25 +0200 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: <51E7C5BD.3080308@gmail.com> References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> <51E7C5BD.3080308@gmail.com> Message-ID: 2013/7/18 Eleytherios Stamatogiannakis > Why is there such a huge speed difference between cPickle.dump( ... f) and > f.write(cPickle.dumps(...)) ? Did you count the number of calls to f.write? pickle call write() once per pickled object. Now, pypy's implementation of buffered file uses a (RPython) list of strings, and does a final ''.join. This is probably much less efficient than the RStringIO implementation. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From estama at gmail.com Thu Jul 18 13:25:28 2013 From: estama at gmail.com (Eleytherios Stamatogiannakis) Date: Thu, 18 Jul 2013 14:25:28 +0300 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> <51E7C5BD.3080308@gmail.com> Message-ID: <51E7D0A8.5050708@gmail.com> Yes you are right, cPickle.dump most probably does a lot more small writes. On the other hand, shouldn't this also affect CPython? Or is CPython so much faster in "".join-ing strings? Is "".join-ing something that we should generally avoid doing in PyPy? l. On 18/07/13 13:53, Amaury Forgeot d'Arc wrote: > 2013/7/18 Eleytherios Stamatogiannakis > > > Why is there such a huge speed difference between cPickle.dump( ... > f) and f.write(cPickle.dumps(...)) ? > > > Did you count the number of calls to f.write? > pickle call write() once per pickled object. > > Now, pypy's implementation of buffered file uses a (RPython) list of > strings, and does a final ''.join. > This is probably much less efficient than the RStringIO implementation. > > -- > Amaury Forgeot d'Arc From amauryfa at gmail.com Thu Jul 18 13:38:46 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 18 Jul 2013 13:38:46 +0200 Subject: [pypy-dev] PyPy 2x slower using cpickle In-Reply-To: <51E7D0A8.5050708@gmail.com> References: <51D42F72.3030501@wyplay.com> <51D44CA2.1040803@gmail.com> <51E7C5BD.3080308@gmail.com> <51E7D0A8.5050708@gmail.com> Message-ID: 2013/7/18 Eleytherios Stamatogiannakis > Yes you are right, cPickle.dump most probably does a lot more small writes. > > On the other hand, shouldn't this also affect CPython? Or is CPython so > much faster in "".join-ing strings? > CPython is not affected, because files are implemented with the C fopen, fwrite... and use a completely different buffer. > Is "".join-ing something that we should generally avoid doing in PyPy? > This is in RPython code, not in the PyPy interpreter. But yes, a buffering method that is optimized for append() + flush() is better than an all-purpose object. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars.wassermann at student.hpi.uni-potsdam.de Fri Jul 19 11:36:23 2013 From: lars.wassermann at student.hpi.uni-potsdam.de (Lars Wassermann) Date: Fri, 19 Jul 2013 11:36:23 +0200 Subject: [pypy-dev] asmgcc assertion error in lang-smalltalk Message-ID: <51E90897.6080903@student.hpi.uni-potsdam.de> Hello, in the project lang-smalltalk, we have a problem we don't know where exactly the source may be. When running the VM with jit, we get: PyPy assertion failed at rpython_memory_gctransform_asmgcroot.c:583: in pypy_g_locate_caller_based_on_retaddr: found a stack frame that does not belong anywhere I know, bug in asmgcc Aborted (core dumped) When running without jit, everything works fine. Is the fact that the error only occurs when using the JIT a hint that the bug may not be in asmgcc? The code to interact with plugins which where compiled as dynamic libraries for the original Smalltalk VM is there [1]. When loading a plugin, it is initialized supplying a (named) function list, which is created around line 978. Can anybody see an obvious mistake we did when using the llhelper? (The llhelper-arguments are recorded in line 104.) Thank you, Lars [1] https://bitbucket.org/pypy/lang-smalltalk/src/8f27fbc5ff8c38b44be7ccbfa060cfc27e6dc3d8/spyvm/interpreter_proxy.py From cfbolz at gmx.de Fri Jul 19 11:51:42 2013 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 19 Jul 2013 11:51:42 +0200 Subject: [pypy-dev] asmgcc assertion error in lang-smalltalk In-Reply-To: <51E90897.6080903@student.hpi.uni-potsdam.de> References: <51E90897.6080903@student.hpi.uni-potsdam.de> Message-ID: <51E90C2E.1050401@gmx.de> On 19/07/13 11:36, Lars Wassermann wrote: > Hello, > in the project lang-smalltalk, we have a problem we don't know where > exactly the source may be. > When running the VM with jit, we get: > > PyPy assertion failed at rpython_memory_gctransform_asmgcroot.c:583: > in pypy_g_locate_caller_based_on_retaddr: found a stack frame that does > not belong anywhere I know, bug in asmgcc > Aborted (core dumped) > > When running without jit, everything works fine. > > Is the fact that the error only occurs when using the JIT a hint that > the bug may not be in asmgcc? You can try to see whether it goes away when not using asmgcc, by translating with: rpython --gcrootfinder=shadowstack -Ojit ...target... Cheers, Carl Friedrich From lars.wassermann at student.hpi.uni-potsdam.de Fri Jul 19 12:28:40 2013 From: lars.wassermann at student.hpi.uni-potsdam.de (Lars Wassermann) Date: Fri, 19 Jul 2013 12:28:40 +0200 Subject: [pypy-dev] asmgcc assertion error in lang-smalltalk In-Reply-To: <51E90C2E.1050401@gmx.de> References: <51E90897.6080903@student.hpi.uni-potsdam.de> <51E90C2E.1050401@gmx.de> Message-ID: <51E914D8.8050205@student.hpi.uni-potsdam.de> On 07/19/2013 11:51 AM, Carl Friedrich Bolz wrote: > On 19/07/13 11:36, Lars Wassermann wrote: >> Hello, >> in the project lang-smalltalk, we have a problem we don't know where >> exactly the source may be. >> When running the VM with jit, we get: >> >> PyPy assertion failed at rpython_memory_gctransform_asmgcroot.c:583: >> in pypy_g_locate_caller_based_on_retaddr: found a stack frame that does >> not belong anywhere I know, bug in asmgcc >> Aborted (core dumped) >> >> When running without jit, everything works fine. >> >> Is the fact that the error only occurs when using the JIT a hint that >> the bug may not be in asmgcc? > > You can try to see whether it goes away when not using asmgcc, by > translating with: > > rpython --gcrootfinder=shadowstack -Ojit ...target... > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > That fixed the problem. Is there a way to generate a bug report for asmgcc, or would that be too complicated? Thank you, Lars From cfbolz at gmx.de Fri Jul 19 12:44:39 2013 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 19 Jul 2013 12:44:39 +0200 Subject: [pypy-dev] asmgcc assertion error in lang-smalltalk In-Reply-To: <51E914D8.8050205@student.hpi.uni-potsdam.de> References: <51E90897.6080903@student.hpi.uni-potsdam.de> <51E90C2E.1050401@gmx.de> <51E914D8.8050205@student.hpi.uni-potsdam.de> Message-ID: <51E91897.3080402@gmx.de> On 19/07/13 12:28, Lars Wassermann wrote: > On 07/19/2013 11:51 AM, Carl Friedrich Bolz wrote: >> You can try to see whether it goes away when not using asmgcc, by >> translating with: >> >> rpython --gcrootfinder=shadowstack -Ojit ...target... > > That fixed the problem. Is there a way to generate a bug report for > asmgcc, or would that be too complicated? What do you mean? You can add one to bugs.pypy.org (asmgcc is part of PyPy). Cheers, Carl Friedrich From cfbolz at gmx.de Fri Jul 19 15:47:35 2013 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 19 Jul 2013 15:47:35 +0200 Subject: [pypy-dev] Announcement: PyPy London Sprint (August 26 - September 1 2013) Message-ID: <51E94377.2060006@gmx.de> ===================================================================== PyPy London Sprint (August 26 - September 1 2013) ===================================================================== The next PyPy sprint will be in London, United Kingdom for the first time. This is a fully public sprint. PyPy sprints are a very good way to get into PyPy development and no prior PyPy knowledge is necessary. ------------------------------ Goals and topics of the sprint ------------------------------ For newcomers: * bring your application/library and we'll help you port it to PyPy, benchmark and profile * come and write your favorite missing numpy function * help us work on developer tools like jitviewer We'll also work on: * refactoring the JIT optimizations * STM and STM-related topics * anything else attendees are interested in ----------- Exact times ----------- The work days should be August 26 - September 1 2013 (Monday-Sunday). The official plans are for people to arrive on the 26th, and to leave on the 2nd. There will be a break day in the middle. We'll typically start at 10:00 in the morning. ------------ Location ------------ The sprint will happen within a room of `King's College's`_ `Strand Campus`_ in `Central London, UK`_. There are some travel instructions `how to get there`_. We are being hosted by `Laurence Tratt`_ and the `Software Development Team`_. .. _`King's College's`: http://www.kcl.ac.uk/ .. _`Central London, UK`: http://goo.gl/maps/Qz0zz .. _`Strand Campus`: http://www.kcl.ac.uk/campuslife/campuses/strand/StrandCampusLocation.aspx .. _`how to get there`: http://www.kcl.ac.uk/campuslife/campuses/directions/strand.aspx .. _`Laurence Tratt`: http://tratt.net/laurie .. _`Software Development Team`: http://soft-dev.org ------------ Demo Session ------------ If you don't want to come to the full sprint, but still want to chat a bit, we are planning to have a demo session on Tuesday August 27. We will announce this separately on the blog. If you are interested, please leave a comment. -------------- Registration -------------- If you want to attend, please register by adding yourself to the "people.txt" file in Mercurial:: https://bitbucket.org/pypy/extradoc/ https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/london-2013 or on the pypy-dev mailing list if you do not yet have check-in rights:: http://mail.python.org/mailman/listinfo/pypy-dev Remember that you may need a (insert country here)-to-UK power adapter. Please note that UK is not within the Schengen zone, so non-EU and non-Switzerland citizens may require specific visa. Please check `travel regulations`_. Also, the UK uses pound sterling (GBP). .. _`travel regulations`: http://www.ukba.homeoffice.gov.uk/visas-immigration/do-you-need-a-visa/ From arigo at tunes.org Sat Jul 20 09:32:31 2013 From: arigo at tunes.org (Armin Rigo) Date: Sat, 20 Jul 2013 09:32:31 +0200 Subject: [pypy-dev] asmgcc assertion error in lang-smalltalk In-Reply-To: <51E90897.6080903@student.hpi.uni-potsdam.de> References: <51E90897.6080903@student.hpi.uni-potsdam.de> Message-ID: Hi Lars, On Fri, Jul 19, 2013 at 11:36 AM, Lars Wassermann > PyPy assertion failed at rpython_memory_gctransform_asmgcroot.c:583: > in pypy_g_locate_caller_based_on_retaddr: found a stack frame that does > not belong anywhere I know, bug in asmgcc The best report for us would be to include exactly on which precise OS this occurs, with which exact version of GCC, and the exact source code (url and revision number); and the example that crashes. Then we hope that the bug is really reproducible, which might not be the case. You can also try to give us the binary, and we attempt to run it. Failing that, you need to debug yourself the crash, which I can explain how to do (but is not trivial). A bient?t, Armin. From kunal.r.puri at gmail.com Sat Jul 20 18:20:03 2013 From: kunal.r.puri at gmail.com (Kunal Puri) Date: Sat, 20 Jul 2013 21:50:03 +0530 Subject: [pypy-dev] pypy-dev Digest, Vol 27, Issue 14 In-Reply-To: References: Message-ID: Well i would love to meet you so do let me know. I came home this evening amidst heavy rain and all. The weather here just seems to get worse each year. Was on my way back when i read the news. Your provine or not. I think its terrible when circumstances push you to such an extreme to do such a thing. 105 days! Wow. Thats a long time. I hope it isnt much longer now. Love you babe..and ill speak to you on mon... On Jul 20, 2013 3:32 PM, wrote: > Send pypy-dev mailing list submissions to > pypy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/pypy-dev > or, via email, send a message with subject or body 'help' to > pypy-dev-request at python.org > > You can reach the person managing the list at > pypy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of pypy-dev digest..." > > > Today's Topics: > > 1. Re: asmgcc assertion error in lang-smalltalk (Lars Wassermann) > 2. Re: asmgcc assertion error in lang-smalltalk (Carl Friedrich Bolz) > 3. Announcement: PyPy London Sprint (August 26 - September 1 > 2013) (Carl Friedrich Bolz) > 4. Re: asmgcc assertion error in lang-smalltalk (Armin Rigo) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 19 Jul 2013 12:28:40 +0200 > From: Lars Wassermann > To: > Subject: Re: [pypy-dev] asmgcc assertion error in lang-smalltalk > Message-ID: <51E914D8.8050205 at student.hpi.uni-potsdam.de> > Content-Type: text/plain; charset="ISO-8859-1" > > On 07/19/2013 11:51 AM, Carl Friedrich Bolz wrote: > > On 19/07/13 11:36, Lars Wassermann wrote: > >> Hello, > >> in the project lang-smalltalk, we have a problem we don't know where > >> exactly the source may be. > >> When running the VM with jit, we get: > >> > >> PyPy assertion failed at rpython_memory_gctransform_asmgcroot.c:583: > >> in pypy_g_locate_caller_based_on_retaddr: found a stack frame that does > >> not belong anywhere I know, bug in asmgcc > >> Aborted (core dumped) > >> > >> When running without jit, everything works fine. > >> > >> Is the fact that the error only occurs when using the JIT a hint that > >> the bug may not be in asmgcc? > > > > You can try to see whether it goes away when not using asmgcc, by > > translating with: > > > > rpython --gcrootfinder=shadowstack -Ojit ...target... > > > > Cheers, > > > > Carl Friedrich > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev > > > That fixed the problem. Is there a way to generate a bug report for > asmgcc, or would that be too complicated? > > Thank you, > Lars > > > ------------------------------ > > Message: 2 > Date: Fri, 19 Jul 2013 12:44:39 +0200 > From: Carl Friedrich Bolz > To: pypy-dev at python.org > Subject: Re: [pypy-dev] asmgcc assertion error in lang-smalltalk > Message-ID: <51E91897.3080402 at gmx.de> > Content-Type: text/plain; charset=ISO-8859-1 > > On 19/07/13 12:28, Lars Wassermann wrote: > > On 07/19/2013 11:51 AM, Carl Friedrich Bolz wrote: > >> You can try to see whether it goes away when not using asmgcc, by > >> translating with: > >> > >> rpython --gcrootfinder=shadowstack -Ojit ...target... > > > > That fixed the problem. Is there a way to generate a bug report for > > asmgcc, or would that be too complicated? > > What do you mean? You can add one to bugs.pypy.org (asmgcc is part of > PyPy). > > Cheers, > > Carl Friedrich > > > > ------------------------------ > > Message: 3 > Date: Fri, 19 Jul 2013 15:47:35 +0200 > From: Carl Friedrich Bolz > To: pypy-dev at python.org > Subject: [pypy-dev] Announcement: PyPy London Sprint (August 26 - > September 1 2013) > Message-ID: <51E94377.2060006 at gmx.de> > Content-Type: text/plain; charset=ISO-8859-1 > > ===================================================================== > PyPy London Sprint (August 26 - September 1 2013) > ===================================================================== > > The next PyPy sprint will be in London, United Kingdom for the first > time. This is a fully public sprint. PyPy sprints are a very good way > to get into PyPy development and no prior PyPy knowledge is necessary. > > > ------------------------------ > Goals and topics of the sprint > ------------------------------ > > For newcomers: > > * bring your application/library and we'll help you port it to PyPy, > benchmark and profile > > * come and write your favorite missing numpy function > > * help us work on developer tools like jitviewer > > We'll also work on: > > * refactoring the JIT optimizations > > * STM and STM-related topics > > * anything else attendees are interested in > > ----------- > Exact times > ----------- > > The work days should be August 26 - September 1 2013 (Monday-Sunday). > The official plans are for people to arrive on the 26th, and > to leave on the 2nd. There will be a break day in the middle. > We'll typically start at 10:00 in the morning. > > > ------------ > Location > ------------ > > The sprint will happen within a room of `King's College's`_ `Strand > Campus`_ in `Central London, UK`_. There are some travel instructions > `how to > get there`_. We are being hosted by `Laurence Tratt`_ and the `Software > Development Team`_. > > .. _`King's College's`: http://www.kcl.ac.uk/ > .. _`Central London, UK`: http://goo.gl/maps/Qz0zz > .. _`Strand Campus`: > http://www.kcl.ac.uk/campuslife/campuses/strand/StrandCampusLocation.aspx > .. _`how to get there`: > http://www.kcl.ac.uk/campuslife/campuses/directions/strand.aspx > .. _`Laurence Tratt`: http://tratt.net/laurie > .. _`Software Development Team`: http://soft-dev.org > > ------------ > Demo Session > ------------ > > If you don't want to come to the full sprint, but still want to chat a > bit, we are planning to have a demo session on Tuesday August 27. We > will announce this separately on the blog. If you are interested, please > leave a comment. > > -------------- > Registration > -------------- > > If you want to attend, please register by adding yourself to the > "people.txt" file in Mercurial:: > > https://bitbucket.org/pypy/extradoc/ > https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/london-2013 > > or on the pypy-dev mailing list if you do not yet have check-in rights:: > > http://mail.python.org/mailman/listinfo/pypy-dev > > Remember that you may need a (insert country here)-to-UK power adapter. > Please note that UK is not within the Schengen zone, so non-EU and > non-Switzerland citizens may require specific visa. Please check `travel > regulations`_. Also, the UK uses pound sterling (GBP). > > .. _`travel regulations`: > http://www.ukba.homeoffice.gov.uk/visas-immigration/do-you-need-a-visa/ > > > ------------------------------ > > Message: 4 > Date: Sat, 20 Jul 2013 09:32:31 +0200 > From: Armin Rigo > To: Lars Wassermann > Cc: PyPy Developer Mailing List > Subject: Re: [pypy-dev] asmgcc assertion error in lang-smalltalk > Message-ID: > xwvU+KStJOxL88ja3Q at mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Hi Lars, > > On Fri, Jul 19, 2013 at 11:36 AM, Lars Wassermann > > PyPy assertion failed > at rpython_memory_gctransform_asmgcroot.c:583: > > in pypy_g_locate_caller_based_on_retaddr: found a stack frame that does > > not belong anywhere I know, bug in asmgcc > > The best report for us would be to include exactly on which precise OS > this occurs, with which exact version of GCC, and the exact source > code (url and revision number); and the example that crashes. Then we > hope that the bug is really reproducible, which might not be the case. > > You can also try to give us the binary, and we attempt to run it. > > Failing that, you need to debug yourself the crash, which I can > explain how to do (but is not trivial). > > > A bient?t, > > Armin. > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > > ------------------------------ > > End of pypy-dev Digest, Vol 27, Issue 14 > **************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars.wassermann at student.hpi.uni-potsdam.de Sun Jul 21 01:51:21 2013 From: lars.wassermann at student.hpi.uni-potsdam.de (Lars Wassermann) Date: Sun, 21 Jul 2013 01:51:21 +0200 Subject: [pypy-dev] asmgcc assertion error in lang-smalltalk In-Reply-To: References: <51E90897.6080903@student.hpi.uni-potsdam.de> Message-ID: <51EB2279.1090107@student.hpi.uni-potsdam.de> Hi Armin, the problem occured on a Ubuntu x86-64 in a x86 chroot with gcc --version: gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 If not translated with --lldebug, you would just get a Segmentation fault. Nevertheless, I was not able to reproduce that behavior. Even when compiling old versions, There are too many other problems still left in the project. Or maybe the problem was solved by an update of gcc. For now, I will again compile with asmgcc and if the problem occurs again, I will report it. Best, Lars On 07/20/2013 09:32 AM, Armin Rigo wrote: > Hi Lars, > > On Fri, Jul 19, 2013 at 11:36 AM, Lars Wassermann > > PyPy assertion > failed at rpython_memory_gctransform_asmgcroot.c:583: >> in pypy_g_locate_caller_based_on_retaddr: found a stack frame >> that does not belong anywhere I know, bug in asmgcc > > The best report for us would be to include exactly on which precise > OS this occurs, with which exact version of GCC, and the exact > source code (url and revision number); and the example that > crashes. Then we hope that the bug is really reproducible, which > might not be the case. > > You can also try to give us the binary, and we attempt to run it. > > Failing that, you need to debug yourself the crash, which I can > explain how to do (but is not trivial). > > > A bient?t, > > Armin. > From njh at njhurst.com Mon Jul 22 05:08:13 2013 From: njh at njhurst.com (Nathan Hurst) Date: Mon, 22 Jul 2013 13:08:13 +1000 Subject: [pypy-dev] parallel building Message-ID: <20130722030813.GA19653@ajhurst.org> Dear mailing list, I have looked around the web and through the various makefiles and pypy/goal/targetpypystandalone.py and I can't work out how to make pypy build using more cores. I have plenty of RAM (64GB) but the build is only using a single core. Is it possible? njh From alex.gaynor at gmail.com Mon Jul 22 05:25:46 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 21 Jul 2013 20:25:46 -0700 Subject: [pypy-dev] parallel building In-Reply-To: <20130722030813.GA19653@ajhurst.org> References: <20130722030813.GA19653@ajhurst.org> Message-ID: No, there currently isn't a way to parallelize building. Alex PS: Attentive readers will note that technically the very last phase of compilation is parallelized. On Sun, Jul 21, 2013 at 8:08 PM, Nathan Hurst wrote: > Dear mailing list, > I have looked around the web and through the various makefiles and > pypy/goal/targetpypystandalone.py and I can't work out how to make > pypy build using more cores. I have plenty of RAM (64GB) but the > build is only using a single core. Is it possible? > > njh > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From njh at njhurst.com Mon Jul 22 08:18:52 2013 From: njh at njhurst.com (Nathan Hurst) Date: Mon, 22 Jul 2013 16:18:52 +1000 Subject: [pypy-dev] parallel building In-Reply-To: References: <20130722030813.GA19653@ajhurst.org> Message-ID: <20130722061852.GB20606@ajhurst.org> On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote: > No, there currently isn't a way to parallelize building. Ok. Is it hard or just low priority? > Alex > > PS: Attentive readers will note that technically the very last phase of > compilation is parallelized. Yes, and I guess if it weren't it would actually take a significant chunk of the compile time :) njh From fijall at gmail.com Mon Jul 22 09:06:51 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 22 Jul 2013 09:06:51 +0200 Subject: [pypy-dev] parallel building In-Reply-To: <20130722061852.GB20606@ajhurst.org> References: <20130722030813.GA19653@ajhurst.org> <20130722061852.GB20606@ajhurst.org> Message-ID: On Mon, Jul 22, 2013 at 8:18 AM, Nathan Hurst wrote: > On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote: >> No, there currently isn't a way to parallelize building. > > Ok. Is it hard or just low priority? hard. there is some work ongoing, but we need a working STM first ;-) > >> Alex >> >> PS: Attentive readers will note that technically the very last phase of >> compilation is parallelized. > > Yes, and I guess if it weren't it would actually take a significant > chunk of the compile time :) > > njh > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From william.leslie.ttg at gmail.com Mon Jul 22 09:07:43 2013 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Mon, 22 Jul 2013 17:07:43 +1000 Subject: [pypy-dev] parallel building In-Reply-To: <20130722061852.GB20606@ajhurst.org> References: <20130722030813.GA19653@ajhurst.org> <20130722061852.GB20606@ajhurst.org> Message-ID: On 22 July 2013 16:18, Nathan Hurst wrote: > On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote: >> No, there currently isn't a way to parallelize building. > > Ok. Is it hard or just low priority? It's mostly pretty hard, although it's not the same reason through each of the stages involved - the short answer is that translation mostly involves walking over graphs, including the inter-procedural call graph, and propagating information. In one stage the graph is known mostly ahead of time, but otherwise it's being built as it goes, which means the job list probably stays pretty small. If we managed to make that stage parallel, we'd probably lose out on the fact that several of the steps work by mutating the model, so we'd either need a concurrency model that dealt with that or separate compilation to work. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement. From 6 at egesfewf.mygbiz.com Thu Jul 25 23:17:39 2013 From: 6 at egesfewf.mygbiz.com (GPS tracker with multi discrete) Date: Thu, 25 Jul 2013 21:17:39 +0000 Subject: [pypy-dev] =?gb2312?b?R1BTIHRyYWNrZXIgd2l0aCBtdWx0aSBkaXNjcmV0?= =?gb2312?b?ZdPrxPq5ss/twcvP4LLhoaM=?= Message-ID: <047d7bacbef6529e4004e25c8e85@google.com> Tips: GPS tracker with multi discrete input and output /Attn: purchase manager Dear Sir This is Anna,the sales manager of Redview GPS in China. VT310 is a GPS tracker with 5 discrete inputs ,5 discrete outputs and 2 analog ports . With VT310,you can get vehicle windows status, door status, engine status, temperature and tank fuel level ,etc. This is widely used in truck tracking application. I would appreciate if you forward this letter to Technical Manager or to other expert responsible for technical integration of new products in your company, or provide me with his contact for we could discuss all the details of our future cooperation. Your early reply is highly appreciated. Best Regards Anna https://picasaweb.google.com/lh/sredir?uname=110555139617906055901&target=ALBUM&id=5902607092932410545&authkey=Gv1sRgCIqn9aTk8vidAQ&invite=COjXiY8L&feat=email -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: picasaweblogo-zh_CN.gif Type: image/gif Size: 1646 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: email.jpg Type: image/jpeg Size: 7003 bytes Desc: not available URL: From ronan.lamy at gmail.com Sun Jul 28 15:02:33 2013 From: ronan.lamy at gmail.com (Ronan Lamy) Date: Sun, 28 Jul 2013 14:02:33 +0100 Subject: [pypy-dev] Killing OOType? (was Re: Translating pypy on FreeBSD with CLI backend) In-Reply-To: References: <1507130.Uoj2OIuUtg@dragon.dg> <1725112.nlyumnnUK3@dragon.dg> <518AC2E9.6010200@gmx.de> <518B57C5.10001@gmail.com> Message-ID: <51F51669.4010704@gmail.com> Le 07/07/13 00:42, Alex Gaynor a ?crit : > Yes, I think we can kill it (if you want some help deleting code, let us > know :P) > The branch has just been merged. However, there's some more cleanup to do, chiefly getting rid of the type_system abstraction. I've started looking into it, it seems that rpython/rtyper/lltypesystem/ can be removed entirely with its contents merged with rpython/rtyper/. > > On Sun, Jul 7, 2013 at 9:39 AM, Ronan Lamy > wrote: > > 2013/5/9 Armin Rigo > > > Hi all, > > On Thu, May 9, 2013 at 10:01 AM, Antonio Cuni > > wrote: > > Although I have an emotional feeling with that piece of code, > I think that > > Alex is right. > > I also tend to agree. Killing stuff that nobody seriously cares > about > is sad but good, particularly when it adds some > otherwise-unnecessary > levels of abstractions everywhere. We should ideally wait e.g. one > month for feedback from other developers that may still have plans > there. > > > > It's been almost 2 months. Can we kill it *now*? > > I've just started a branch: > https://bitbucket.org/pypy/pypy/commits/branch/kill-ootype > It looks like killing it properly requires some work. From fijall at gmail.com Sun Jul 28 21:25:48 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 28 Jul 2013 21:25:48 +0200 Subject: [pypy-dev] Killing OOType? (was Re: Translating pypy on FreeBSD with CLI backend) In-Reply-To: <51F51669.4010704@gmail.com> References: <1507130.Uoj2OIuUtg@dragon.dg> <1725112.nlyumnnUK3@dragon.dg> <518AC2E9.6010200@gmx.de> <518B57C5.10001@gmail.com> <51F51669.4010704@gmail.com> Message-ID: On Sun, Jul 28, 2013 at 3:02 PM, Ronan Lamy wrote: > Le 07/07/13 00:42, Alex Gaynor a ?crit : > >> Yes, I think we can kill it (if you want some help deleting code, let us >> know :P) >> > The branch has just been merged. > > However, there's some more cleanup to do, chiefly getting rid of the > type_system abstraction. I've started looking into it, it seems that > rpython/rtyper/lltypesystem/ can be removed entirely with its contents > merged with rpython/rtyper/. > >> >> On Sun, Jul 7, 2013 at 9:39 AM, Ronan Lamy > > wrote: >> >> 2013/5/9 Armin Rigo > >> >> >> Hi all, >> >> On Thu, May 9, 2013 at 10:01 AM, Antonio Cuni >> > wrote: >> > Although I have an emotional feeling with that piece of code, >> I think that >> > Alex is right. >> >> I also tend to agree. Killing stuff that nobody seriously cares >> about >> is sad but good, particularly when it adds some >> otherwise-unnecessary >> levels of abstractions everywhere. We should ideally wait e.g. >> one >> month for feedback from other developers that may still have plans >> there. >> >> >> >> It's been almost 2 months. Can we kill it *now*? >> >> I've just started a branch: >> https://bitbucket.org/pypy/pypy/commits/branch/kill-ootype >> It looks like killing it properly requires some work. > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev Hi ronan Good work! Can you write some info to the mailing list summarizing the discussion and the decision? Cheers, fijal From njh at njhurst.com Mon Jul 29 01:28:08 2013 From: njh at njhurst.com (Nathan Hurst) Date: Mon, 29 Jul 2013 09:28:08 +1000 Subject: [pypy-dev] pypy ignores signals sometimes was: parallel building In-Reply-To: References: <20130722030813.GA19653@ajhurst.org> <20130722061852.GB20606@ajhurst.org> Message-ID: <20130728232808.GA12813@ajhurst.org> On Mon, Jul 22, 2013 at 09:06:51AM +0200, Maciej Fijalkowski wrote: > On Mon, Jul 22, 2013 at 8:18 AM, Nathan Hurst wrote: > > On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote: > >> No, there currently isn't a way to parallelize building. > > > > Ok. Is it hard or just low priority? > > hard. there is some work ongoing, but we need a working STM first ;-) Thanks for the explanation Alex+Maciej - I'll just get used to waiting. Ok. I'm just learning about python threading (after reading about STMs - I've always used multiprocessing for compute heavy stuff and greenlets for servery stuff) and I tried this simple program: import threading class MyThread(threading.Thread): def run(self): while True: pass thread = MyThread() thread.start() thread.join() now clearly this program is never going to terminate without some kind of external interrupt. But the strange thing is, it doesn't terminate when I use control-C on it either (it does terminate correctly from eclipse.pydev with the stop button): njh $ pypy threadtoy.py ^C ^Z [1]+ Stopped pypy threadtoy.py njh $ kill % [1]+ Stopped pypy threadtoy.py njh $ [1]+ Terminated pypy threadtoy.py why does python and pypy do this? Is it part of the GIL problem? will STM fix it? regards, njh From arigo at tunes.org Mon Jul 29 09:18:53 2013 From: arigo at tunes.org (Armin Rigo) Date: Mon, 29 Jul 2013 09:18:53 +0200 Subject: [pypy-dev] pypy ignores signals sometimes was: parallel building In-Reply-To: <20130728232808.GA12813@ajhurst.org> References: <20130722030813.GA19653@ajhurst.org> <20130722061852.GB20606@ajhurst.org> <20130728232808.GA12813@ajhurst.org> Message-ID: Hi Nathan, On Mon, Jul 29, 2013 at 1:28 AM, Nathan Hurst wrote: > why does python and pypy do this? Is it part of the GIL problem? > will STM fix it? PyPy does this because Python does it. It is not part of any other problem, just that thread.join() uses lock.acquire(), which cannot be interrupted in Python 2.x. This has been changed in some version of Python 3.x. A bient?t, Armin. From njh at njhurst.com Tue Jul 30 02:43:23 2013 From: njh at njhurst.com (Nathan Hurst) Date: Tue, 30 Jul 2013 10:43:23 +1000 Subject: [pypy-dev] memory leak in pypy Message-ID: <20130730004323.GA6029@ajhurst.org> I was playing with this simple function to compute uint/3. It does not (afaict) directly allocate any memory, but when run it rapidly consumes all memory (32GB): def divu3(n): q = (n >> 2) + (n >> 4) # q = n*0.0101 (approx). q = q + (q >> 4) # q = n*0.01010101. q = q + (q >> 8) # q = n*0.01010101. q = q + (q >> 16) # q = n*0.01010101. r = n - q*3 # 0 <= r <= 15. return q + (11*r >> 5) # Returning q + r/3. for i in range(2**31): assert(divu3(i) == i/3) Python 2.7.3 (daf1b0412bfbd0666c19d567e37b29e4a3be5734, Jul 12 2013, 19:10:57) [PyPy 2.1.0-beta1 with GCC 4.7.2] on linux2 is it being over eager to specialise? njh From njh at njhurst.com Tue Jul 30 02:15:09 2013 From: njh at njhurst.com (Nathan Hurst) Date: Tue, 30 Jul 2013 10:15:09 +1000 Subject: [pypy-dev] intrinsics for special instructions Message-ID: <20130730001509.GA5977@ajhurst.org> Is there a way to call special operators such as __builtin_popcount from within pypy? regards, njh From alex.gaynor at gmail.com Tue Jul 30 02:56:10 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 29 Jul 2013 17:56:10 -0700 Subject: [pypy-dev] intrinsics for special instructions In-Reply-To: <20130730001509.GA5977@ajhurst.org> References: <20130730001509.GA5977@ajhurst.org> Message-ID: There's currently no special way, you could bind to it using cffi; but this will generate a call to a C function, not the instruction directly. Alex On Mon, Jul 29, 2013 at 5:15 PM, Nathan Hurst wrote: > Is there a way to call special operators such as __builtin_popcount > from within pypy? > > regards, > njh > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Jul 30 10:57:43 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 30 Jul 2013 10:57:43 +0200 Subject: [pypy-dev] Fwd: Build failed in Jenkins: pypy #887 In-Reply-To: <1390320528.23.1375145335413.JavaMail.jenkins@ci> References: <689422487.22.1375123678160.JavaMail.jenkins@ci> <1390320528.23.1375145335413.JavaMail.jenkins@ci> Message-ID: the nightly build crashes strangely ---------- Forwarded message ---------- From: "Jenkins" Date: Tue, Jul 30, 2013 at 2:48 AM Subject: Build failed in Jenkins: pypy #887 To: fijall at gmail.com See ------------------------------------------ [...truncated 3127 lines...] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2268:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2268:3: warning: (near initialization for ?pypy_g_array_690.a.items[432].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2283:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2283:3: warning: (near initialization for ?pypy_g_array_690.a.items[435].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2298:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2298:3: warning: (near initialization for ?pypy_g_array_690.a.items[438].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2308:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2308:3: warning: (near initialization for ?pypy_g_array_690.a.items[440].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2313:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2313:3: warning: (near initialization for ?pypy_g_array_690.a.items[441].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2333:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2333:3: warning: (near initialization for ?pypy_g_array_690.a.items[445].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2383:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2383:3: warning: (near initialization for ?pypy_g_array_690.a.items[455].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2478:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2478:3: warning: (near initialization for ?pypy_g_array_690.a.items[474].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2503:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2503:3: warning: (near initialization for ?pypy_g_array_690.a.items[479].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2563:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2563:3: warning: (near initialization for ?pypy_g_array_690.a.items[491].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2578:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2578:3: warning: (near initialization for ?pypy_g_array_690.a.items[494].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2588:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2588:3: warning: (near initialization for ?pypy_g_array_690.a.items[496].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2593:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2593:3: warning: (near initialization for ?pypy_g_array_690.a.items[497].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2613:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2613:3: warning: (near initialization for ?pypy_g_array_690.a.items[501].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2643:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2643:3: warning: (near initialization for ?pypy_g_array_690.a.items[507].d_value?) [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2663:3: warning: initialization from incompatible pointer type [enabled by default] [translation:ERROR] data_pypy_module_cpyext_pyobject.c:2663:3: warning: (near initialization for ?pypy_g_array_690.a.items[511].d_value?) [enabled by default] [translation:ERROR] implement_1.c: In function ?pypy_g_descr_typecheck_get_doc?: [translation:ERROR] implement_1.c:14072:10: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_5.c: In function ?pypy_g_ccall_fclose__arrayPtr_reload?: [translation:ERROR] implement_5.c:15324:2: warning: passing argument 1 of ?fclose? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from :4:0, [translation:ERROR] from :85, [translation:ERROR] from common_header.h:38, [translation:ERROR] from implement_5.c:5: [translation:ERROR] /usr/include/stdio.h:234:12: note: expected ?struct FILE *? but argument is of type ?char *? [translation:ERROR] implement_9.c: In function ?pypy_g_ccall_stat64__arrayPtr_statPtr_reload?: [translation:ERROR] implement_9.c:12377:2: warning: passing argument 2 of ?stat64? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:55:0, [translation:ERROR] from implement_9.c:5: [translation:ERROR] /usr/include/x86_64-linux-gnu/sys/stat.h:504:1: note: expected ?struct stat64 *? but argument is of type ?struct stat *? [translation:ERROR] implement_9.c: In function ?pypy_g_ccall_fstat64__INT_statPtr_reload?: [translation:ERROR] implement_9.c:12617:2: warning: passing argument 2 of ?fstat64? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:55:0, [translation:ERROR] from implement_9.c:5: [translation:ERROR] /usr/include/x86_64-linux-gnu/sys/stat.h:518:1: note: expected ?struct stat64 *? but argument is of type ?struct stat *? [translation:ERROR] implement_11.c: In function ?pypy_g_ccall_lstat64__arrayPtr_statPtr_reload?: [translation:ERROR] implement_11.c:24466:2: warning: passing argument 2 of ?lstat64? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:55:0, [translation:ERROR] from implement_11.c:5: [translation:ERROR] /usr/include/x86_64-linux-gnu/sys/stat.h:511:1: note: expected ?struct stat64 *? but argument is of type ?struct stat *? [translation:ERROR] implement_11.c: In function ?pypy_g_ccall_fdopen__INT_arrayPtr_reload?: [translation:ERROR] implement_11.c:24524:11: warning: assignment from incompatible pointer type [enabled by default] [translation:ERROR] implement_11.c: In function ?pypy_g_ccall_setbuf__arrayPtr_arrayPtr_reload?: [translation:ERROR] implement_11.c:24581:2: warning: passing argument 1 of ?setbuf? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from :4:0, [translation:ERROR] from :85, [translation:ERROR] from common_header.h:38, [translation:ERROR] from implement_11.c:5: [translation:ERROR] /usr/include/stdio.h:329:13: note: expected ?struct FILE * __restrict__? but argument is of type ?char *? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetUnparsedEntityDeclHandler__NonePtr__1?: [translation:ERROR] implement_21.c:24275:2: warning: passing argument 2 of ?XML_SetUnparsedEntityDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:600:1: note: expected ?XML_UnparsedEntityDeclHandler? but argument is of type ?void (*)(void *, char *, char *, char *, char *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetStartElementHandler__NonePtr_funcPt_1?: [translation:ERROR] implement_21.c:24451:2: warning: passing argument 2 of ?XML_SetStartElementHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:539:1: note: expected ?XML_StartElementHandler? but argument is of type ?void (*)(void *, char *, char **)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetSkippedEntityHandler__NonePtr_funcP_1?: [translation:ERROR] implement_21.c:24594:2: warning: passing argument 2 of ?XML_SetSkippedEntityHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:637:1: note: expected ?XML_SkippedEntityHandler? but argument is of type ?void (*)(void *, char *, int)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetEndElementHandler__NonePtr_funcPtr_?: [translation:ERROR] implement_21.c:24664:2: warning: passing argument 2 of ?XML_SetEndElementHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:543:1: note: expected ?XML_EndElementHandler? but argument is of type ?void (*)(void *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetDefaultHandlerExpand__NonePtr_funcP_1?: [translation:ERROR] implement_21.c:24734:2: warning: passing argument 2 of ?XML_SetDefaultHandlerExpand? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:583:1: note: expected ?XML_DefaultHandler? but argument is of type ?void (*)(void *, char *, int)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetCharacterDataHandler__NonePtr_funcP_1?: [translation:ERROR] implement_21.c:24804:2: warning: passing argument 2 of ?XML_SetCharacterDataHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:547:1: note: expected ?XML_CharacterDataHandler? but argument is of type ?void (*)(void *, char *, int)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetCommentHandler__NonePtr_funcPtr_rel?: [translation:ERROR] implement_21.c:24929:2: warning: passing argument 2 of ?XML_SetCommentHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:554:1: note: expected ?XML_CommentHandler? but argument is of type ?void (*)(void *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetElementDeclHandler__NonePtr_funcPtr_1?: [translation:ERROR] implement_21.c:24999:2: warning: passing argument 2 of ?XML_SetElementDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:155:1: note: expected ?XML_ElementDeclHandler? but argument is of type ?void (*)(void *, char *, struct XML_Content *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetStartNamespaceDeclHandler__NonePtr__1?: [translation:ERROR] implement_21.c:25193:2: warning: passing argument 2 of ?XML_SetStartNamespaceDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:613:1: note: expected ?XML_StartNamespaceDeclHandler? but argument is of type ?void (*)(void *, char *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetEndNamespaceDeclHandler__NonePtr_fu_1?: [translation:ERROR] implement_21.c:25263:2: warning: passing argument 2 of ?XML_SetEndNamespaceDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:617:1: note: expected ?XML_EndNamespaceDeclHandler? but argument is of type ?void (*)(void *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetEntityDeclHandler__NonePtr_funcPtr_?: [translation:ERROR] implement_21.c:25334:2: warning: passing argument 2 of ?XML_SetEntityDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:341:1: note: expected ?XML_EntityDeclHandler? but argument is of type ?void (*)(void *, char *, int, char *, int, char *, char *, char *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetNotationDeclHandler__NonePtr_funcPt_1?: [translation:ERROR] implement_21.c:25405:2: warning: passing argument 2 of ?XML_SetNotationDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:604:1: note: expected ?XML_NotationDeclHandler? but argument is of type ?void (*)(void *, char *, char *, char *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetStartDoctypeDeclHandler__NonePtr_fu_1?: [translation:ERROR] implement_21.c:25531:2: warning: passing argument 2 of ?XML_SetStartDoctypeDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:592:1: note: expected ?XML_StartDoctypeDeclHandler? but argument is of type ?void (*)(void *, char *, char *, char *, int)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetDefaultHandler__NonePtr_funcPtr_rel?: [translation:ERROR] implement_21.c:25601:2: warning: passing argument 2 of ?XML_SetDefaultHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:575:1: note: expected ?XML_DefaultHandler? but argument is of type ?void (*)(void *, char *, int)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetExternalEntityRefHandler__NonePtr_f_1?: [translation:ERROR] implement_21.c:25672:2: warning: passing argument 2 of ?XML_SetExternalEntityRefHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:625:1: note: expected ?XML_ExternalEntityRefHandler? but argument is of type ?int (*)(struct XML_ParserStruct *, char *, char *, char *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetProcessingInstructionHandler__NoneP_1?: [translation:ERROR] implement_21.c:25743:2: warning: passing argument 2 of ?XML_SetProcessingInstructionHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:551:1: note: expected ?XML_ProcessingInstructionHandler? but argument is of type ?void (*)(void *, char *, char *)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetXmlDeclHandler__NonePtr_funcPtr_rel?: [translation:ERROR] implement_21.c:25869:2: warning: passing argument 2 of ?XML_SetXmlDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:192:1: note: expected ?XML_XmlDeclHandler? but argument is of type ?void (*)(void *, char *, char *, int)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_SetAttlistDeclHandler__NonePtr_funcPtr_1?: [translation:ERROR] implement_21.c:25940:2: warning: passing argument 2 of ?XML_SetAttlistDeclHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_21.c:5: [translation:ERROR] /usr/include/expat.h:175:1: note: expected ?XML_AttlistDeclHandler? but argument is of type ?void (*)(void *, char *, char *, char *, char *, int)? [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_EVP_get_digestbyname__arrayPtr_reload?: [translation:ERROR] implement_21.c:60561:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_21.c: In function ?pypy_g_ccall_XML_ErrorString__INT_reload?: [translation:ERROR] implement_21.c:65477:11: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_22.c: In function ?pypy_g_ccall_XML_SetUnknownEncodingHandler__NonePtr_fun_1?: [translation:ERROR] implement_22.c:4813:2: warning: passing argument 2 of ?XML_SetUnknownEncodingHandler? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from common_header.h:80:0, [translation:ERROR] from implement_22.c:5: [translation:ERROR] /usr/include/expat.h:641:1: note: expected ?XML_UnknownEncodingHandler? but argument is of type ?int (*)(void *, char *, struct XML_Encoding *)? [translation:ERROR] implement_22.c: In function ?pypy_g_ccall_SSL_get_current_cipher__SSLPtr_reload?: [translation:ERROR] implement_22.c:46343:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_22.c: In function ?pypy_g_ccall_SSL_CIPHER_get_name__SSL_CIPHERPtr_reload?: [translation:ERROR] implement_22.c:46398:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_23.c: In function ?pypy_g_ccall_openpty__arrayPtr_arrayPtr_arrayPtr_arrayP_1?: [translation:ERROR] implement_23.c:32105:2: warning: implicit declaration of function ?openpty? [-Wimplicit-function-declaration] [translation:ERROR] implement_23.c: In function ?pypy_g_ccall_forkpty__arrayPtr_arrayPtr_arrayPtr_arrayP_1?: [translation:ERROR] implement_23.c:34480:2: warning: implicit declaration of function ?forkpty? [-Wimplicit-function-declaration] [translation:ERROR] implement_23.c: In function ?pypy_g_ccall_EVP_DigestFinal__EVP_MD_CTXPtr_arrayPtr_ar_1?: [translation:ERROR] implement_23.c:41912:2: warning: pointer targets in passing argument 2 of ?EVP_DigestFinal? differ in signedness [-Wpointer-sign] [translation:ERROR] In file included from /usr/include/openssl/x509.h:73:0, [translation:ERROR] from /usr/include/openssl/ssl.h:156, [translation:ERROR] from common_header.h:72, [translation:ERROR] from implement_23.c:5: [translation:ERROR] /usr/include/openssl/evp.h:565:5: note: expected ?unsigned char *? but argument is of type ?char *? [translation:ERROR] implement_23.c: In function ?pypy_g_ccall_i2d_X509__X509Ptr_arrayPtr_reload?: [translation:ERROR] implement_23.c:42675:2: warning: passing argument 2 of ?i2d_X509? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from /usr/include/openssl/ssl.h:156:0, [translation:ERROR] from common_header.h:72, [translation:ERROR] from implement_23.c:5: [translation:ERROR] /usr/include/openssl/x509.h:839:1: note: expected ?unsigned char **? but argument is of type ?char **? [translation:ERROR] implement_23.c: In function ?pypy_g_ccall_inet_ntop__INT_arrayPtr_arrayPtr_UINT_relo?: [translation:ERROR] implement_23.c:48311:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_24.c: In function ?pypy_g_ccall_X509V3_EXT_get__arrayPtr_reload?: [translation:ERROR] implement_24.c:27995:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_24.c: In function ?pypy_g_ccall_ASN1_ITEM_ptr__funcPtr_reload?: [translation:ERROR] implement_24.c:28275:12: warning: assignment from incompatible pointer type [enabled by default] [translation:ERROR] implement_24.c: In function ?pypy_g_ccall_ASN1_item_d2i__arrayPtr_arrayPtr_Signed_AS_1?: [translation:ERROR] implement_24.c:28337:2: warning: passing argument 2 of ?ASN1_item_d2i? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from /usr/include/openssl/objects.h:960:0, [translation:ERROR] from /usr/include/openssl/evp.h:94, [translation:ERROR] from /usr/include/openssl/x509.h:73, [translation:ERROR] from /usr/include/openssl/ssl.h:156, [translation:ERROR] from common_header.h:72, [translation:ERROR] from implement_24.c:5: [translation:ERROR] /usr/include/openssl/asn1.h:1088:14: note: expected ?const unsigned char **? but argument is of type ?char **? [translation:ERROR] implement_24.c: In function ?pypy_g_ccall_TLSv1_method____reload?: [translation:ERROR] implement_24.c:33757:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_24.c: In function ?pypy_g_ccall_SSLv3_method____reload?: [translation:ERROR] implement_24.c:34491:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_24.c: In function ?pypy_g_ccall_SSLv23_method____reload?: [translation:ERROR] implement_24.c:34544:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] implement_24.c: In function ?pypy_g_ccall_ASN1_STRING_to_UTF8__arrayPtr_asn1_string__1?: [translation:ERROR] implement_24.c:41507:2: warning: passing argument 1 of ?ASN1_STRING_to_UTF8? from incompatible pointer type [enabled by default] [translation:ERROR] In file included from /usr/include/openssl/objects.h:960:0, [translation:ERROR] from /usr/include/openssl/evp.h:94, [translation:ERROR] from /usr/include/openssl/x509.h:73, [translation:ERROR] from /usr/include/openssl/ssl.h:156, [translation:ERROR] from common_header.h:72, [translation:ERROR] from implement_24.c:5: [translation:ERROR] /usr/include/openssl/asn1.h:1000:5: note: expected ?unsigned char **? but argument is of type ?char **? [translation:ERROR] implement_25.c: In function ?pypy_g_ccall_gai_strerror__INT_reload?: [translation:ERROR] implement_25.c:10714:12: warning: assignment discards ?const? qualifier from pointer target type [enabled by default] [translation:ERROR] pypy_goal_targetpypystandalone.c: In function ?pypy_g_pypy_thread_attach?: [translation:ERROR] pypy_goal_targetpypystandalone.c:3981:2: warning: assignment makes pointer from integer without a cast [enabled by default] [translation:ERROR] In file included from pypy_interpreter_astcompiler_symtable.c:7:0: [translation:ERROR] forwarddecl.h:207028:14: error: stray ?`? in program [translation:ERROR] forwarddecl.h:207028:35: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? before ?y_g_array_45914? [translation:ERROR] make: *** [pypy_interpreter_astcompiler_symtable.s] Error 1 [translation:ERROR] make: *** Waiting for unfinished jobs.... [translation:ERROR] """) [translation] start debugger... > -> raise CompilationError(stdout, stderr) Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel f(*fargs, **fkwds) File "../../rpython/bin/rpython", line 20, in main() File " line 325, in main debug(True) File " line 278, in debug pdb_plus_show.start(tb) File " line 417, in start fn(*args) File " line 25, in post_mortem self.interaction(t.tb_frame, t) File "/opt/pypy_2_beta2/lib-python/2.7/pdb.py", line 210, in interaction self.cmdloop() File "/opt/pypy_2_beta2/lib-python/2.7/cmd.py", line 109, in cmdloop self.preloop() File " line 29, in preloop raise NoTTY("Cannot start the debugger when stdout is captured.") NoTTY: Cannot start the debugger when stdout is captured. Build step 'Execute shell' marked build as failure From fijall at gmail.com Tue Jul 30 11:01:30 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 30 Jul 2013 11:01:30 +0200 Subject: [pypy-dev] memory leak in pypy In-Reply-To: <20130730004323.GA6029@ajhurst.org> References: <20130730004323.GA6029@ajhurst.org> Message-ID: This sounds odd. My PyPy does not leak memory in this example. Can you please double check? On Tue, Jul 30, 2013 at 2:43 AM, Nathan Hurst wrote: > I was playing with this simple function to compute uint/3. It does > not (afaict) directly allocate any memory, but when run it rapidly > consumes all memory (32GB): > > def divu3(n): > q = (n >> 2) + (n >> 4) # q = n*0.0101 (approx). > q = q + (q >> 4) # q = n*0.01010101. > q = q + (q >> 8) # q = n*0.01010101. > q = q + (q >> 16) # q = n*0.01010101. > r = n - q*3 # 0 <= r <= 15. > return q + (11*r >> 5) # Returning q + r/3. > > > for i in range(2**31): > assert(divu3(i) == i/3) > > > Python 2.7.3 (daf1b0412bfbd0666c19d567e37b29e4a3be5734, Jul 12 2013, 19:10:57) > [PyPy 2.1.0-beta1 with GCC 4.7.2] on linux2 > > is it being over eager to specialise? > > njh > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Tue Jul 30 11:38:26 2013 From: arigo at tunes.org (Armin Rigo) Date: Tue, 30 Jul 2013 11:38:26 +0200 Subject: [pypy-dev] Threading in RPython In-Reply-To: References: Message-ID: Hi Sean, On Sun, Jul 7, 2013 at 10:23 PM, Sean Fisk wrote: > Sorry, I didn't make it clear: my project is actually an interpreter for a > stencil-based language that is used for solving partial differential > equations. So I think that RPython is the right tool to an extent. It worked > very well for writing the interpreter (I also used Alex Gaynor's rply). > However, now I would like to parallelize the solver (i.e., the matrix > operations). > > I sounds as if this is where RPython stops being the right tool for the job, > since threading is useful only for concurrency and not speed, as in Python. > Thank you for letting me know before I spent too much time on it! I suppose > I should have guessed from the code I was writing. Sorry for not answering you earlier. The point is that, indeed, you are right and I have no obvious solution to propose. RPython is really designed to write high-level interpreters in, and so it doesn't worry about real multicore usage. (In fact, the STM approach currently worked on is about giving multicore usage anyway, but without changing the basic model of a GIL, which is not what you need.) It all depends on what the performance characteristics of your interpreter are. Do you spend a lot of time in the interpreter itself, or rather in library code? In the latter case, RPython is not the best approach --- e.g. its JIT is useless, and it's a case where multithreading would really help. I'm guessing that your interpreter is somewhere in the middle... > I am seeing two options, on which I welcome comment: > > Continue using RPython and use rffi to make calls to C, where I implement > the solver function for high speed, optionally with threading or > message-passing. I've used pthreads, OpenMP, and MPI a number of times > before, but combining that with rffi might be a little over my head. > > Switch to using PyPy with multiprocessing and/or numpypy. Happily, my code > runs under PyPy and CPython albeit with some RPython libraries and some > RPython-specific workarounds (e.g., boxes). If you want to run several regular Python threads and still get multicore usage, one of the reasonably easy approaches right now would be to write pure Python code containing calls to C functions written with CFFI's verify(). These calls are done with the GIL released (both on CPython and PyPy). If instead you'd rather have a single Python thread driving calls to C functions that internally spawn several threads, then indeed OpenMP or MPI seems better. In this case it's easier to embed into an RPython program with rffi. You basically have to write C code that exposes some simple single-threaded API to the RPython program. You would use threads only internally, on the C side. A bient?t, Armin. From svpcom at gmail.com Tue Jul 30 14:02:54 2013 From: svpcom at gmail.com (Vasily Evseenko) Date: Tue, 30 Jul 2013 16:02:54 +0400 Subject: [pypy-dev] memory leak in pypy In-Reply-To: References: <20130730004323.GA6029@ajhurst.org> Message-ID: <51F7AB6E.9000201@gmail.com> Hi, It seems to be range / xrange issue. range allocates all data in a moment when xrange acts like an iterator. On 07/30/2013 01:01 PM, Maciej Fijalkowski wrote: > This sounds odd. My PyPy does not leak memory in this example. Can you > please double check? > > On Tue, Jul 30, 2013 at 2:43 AM, Nathan Hurst wrote: >> I was playing with this simple function to compute uint/3. It does >> not (afaict) directly allocate any memory, but when run it rapidly >> consumes all memory (32GB): >> >> def divu3(n): >> q = (n >> 2) + (n >> 4) # q = n*0.0101 (approx). >> q = q + (q >> 4) # q = n*0.01010101. >> q = q + (q >> 8) # q = n*0.01010101. >> q = q + (q >> 16) # q = n*0.01010101. >> r = n - q*3 # 0 <= r <= 15. >> return q + (11*r >> 5) # Returning q + r/3. >> >> >> for i in range(2**31): >> assert(divu3(i) == i/3) >> >> >> Python 2.7.3 (daf1b0412bfbd0666c19d567e37b29e4a3be5734, Jul 12 2013, 19:10:57) >> [PyPy 2.1.0-beta1 with GCC 4.7.2] on linux2 >> >> is it being over eager to specialise? >> >> njh >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Tue Jul 30 14:04:47 2013 From: arigo at tunes.org (Armin Rigo) Date: Tue, 30 Jul 2013 14:04:47 +0200 Subject: [pypy-dev] memory leak in pypy In-Reply-To: <51F7AB6E.9000201@gmail.com> References: <20130730004323.GA6029@ajhurst.org> <51F7AB6E.9000201@gmail.com> Message-ID: Hi Vasily, On Tue, Jul 30, 2013 at 2:02 PM, Vasily Evseenko wrote: > It seems to be range / xrange issue. > range allocates all data in a moment when xrange acts like an iterator. Not in PyPy. Armin From seanfisk at gmail.com Tue Jul 30 17:00:55 2013 From: seanfisk at gmail.com (Sean Fisk) Date: Tue, 30 Jul 2013 09:00:55 -0600 Subject: [pypy-dev] Threading in RPython In-Reply-To: References: Message-ID: On Tue, Jul 30, 2013 at 3:38 AM, Armin Rigo wrote: > Hi Sean, > > On Sun, Jul 7, 2013 at 10:23 PM, Sean Fisk wrote: > > Sorry, I didn't make it clear: my project is actually an interpreter for > a > > stencil-based language that is used for solving partial differential > > equations. So I think that RPython is the right tool to an extent. It > worked > > very well for writing the interpreter (I also used Alex Gaynor's rply). > > However, now I would like to parallelize the solver (i.e., the matrix > > operations). > > > > I sounds as if this is where RPython stops being the right tool for the > job, > > since threading is useful only for concurrency and not speed, as in > Python. > > Thank you for letting me know before I spent too much time on it! I > suppose > > I should have guessed from the code I was writing. > > Sorry for not answering you earlier. The point is that, indeed, you > are right and I have no obvious solution to propose. RPython is > really designed to write high-level interpreters in, and so it doesn't > worry about real multicore usage. (In fact, the STM approach > currently worked on is about giving multicore usage anyway, but > without changing the basic model of a GIL, which is not what you > need.) > > It all depends on what the performance characteristics of your > interpreter are. Do you spend a lot of time in the interpreter > itself, or rather in library code? In the latter case, RPython is not > the best approach --- e.g. its JIT is useless, and it's a case where > multithreading would really help. I'm guessing that your interpreter > is somewhere in the middle... > I am in the process of profiling and running benchmarks right now, but that is absolutely something I would need to find out before starting on any parallel solution. > > > I am seeing two options, on which I welcome comment: > > > > Continue using RPython and use rffi to make calls to C, where I implement > > the solver function for high speed, optionally with threading or > > message-passing. I've used pthreads, OpenMP, and MPI a number of times > > before, but combining that with rffi might be a little over my head. > > > > Switch to using PyPy with multiprocessing and/or numpypy. Happily, my > code > > runs under PyPy and CPython albeit with some RPython libraries and some > > RPython-specific workarounds (e.g., boxes). > > If you want to run several regular Python threads and still get > multicore usage, one of the reasonably easy approaches right now would > be to write pure Python code containing calls to C functions written > with CFFI's verify(). These calls are done with the GIL released > (both on CPython and PyPy). > > If instead you'd rather have a single Python thread driving calls to C > functions that internally spawn several threads, then indeed OpenMP or > MPI seems better. In this case it's easier to embed into an RPython > program with rffi. You basically have to write C code that exposes > some simple single-threaded API to the RPython program. You would use > threads only internally, on the C side. > These are both very reasonable suggestions. Thank you for clearly spelling out my best options. Unfortunately, my internship is coming to an end, and I am unsure whether I will get the chance to implement them. If I do implement them, however, I'll make a point to write about my experience so that others can benefit from these solutions. Thank you for the answer, Armin. - Sean > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njh at njhurst.com Tue Jul 30 20:39:12 2013 From: njh at njhurst.com (Nathan Hurst) Date: Wed, 31 Jul 2013 04:39:12 +1000 Subject: [pypy-dev] memory leak in pypy In-Reply-To: References: <20130730004323.GA6029@ajhurst.org> Message-ID: <20130730183912.GA9029@ajhurst.org> Ok, I tracked it down. For some reason pydev (eclipse) was randomly choosing between pypy and python2.7 on each run (I discovered this when watching what was happening in top). This explains why it was so flakey. Sorry for the confusion. I have no idea how I'm going to debug this further. (OT: does anyone recommend a better IDE than pydev+eclipse?) njh On Tue, Jul 30, 2013 at 11:01:30AM +0200, Maciej Fijalkowski wrote: > This sounds odd. My PyPy does not leak memory in this example. Can you > please double check? > > On Tue, Jul 30, 2013 at 2:43 AM, Nathan Hurst wrote: > > I was playing with this simple function to compute uint/3. It does > > not (afaict) directly allocate any memory, but when run it rapidly > > consumes all memory (32GB): > > > > def divu3(n): > > q = (n >> 2) + (n >> 4) # q = n*0.0101 (approx). > > q = q + (q >> 4) # q = n*0.01010101. > > q = q + (q >> 8) # q = n*0.01010101. > > q = q + (q >> 16) # q = n*0.01010101. > > r = n - q*3 # 0 <= r <= 15. > > return q + (11*r >> 5) # Returning q + r/3. > > > > > > for i in range(2**31): > > assert(divu3(i) == i/3) > > > > > > Python 2.7.3 (daf1b0412bfbd0666c19d567e37b29e4a3be5734, Jul 12 2013, 19:10:57) > > [PyPy 2.1.0-beta1 with GCC 4.7.2] on linux2 > > > > is it being over eager to specialise? > > > > njh > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > http://mail.python.org/mailman/listinfo/pypy-dev From pjenvey at underboss.org Tue Jul 30 22:38:52 2013 From: pjenvey at underboss.org (Philip Jenvey) Date: Tue, 30 Jul 2013 13:38:52 -0700 Subject: [pypy-dev] PyPy3 2.1 beta 1 released Message-ID: ================ PyPy3 2.1 beta 1 ================ We're pleased to announce the first beta of the upcoming 2.1 release of PyPy3. This is the first release of PyPy which targets Python 3 (3.2.3) compatibility. We would like to thank all of the people who donated_ to the `py3k proposal`_ for supporting the work that went into this and future releases. You can download the PyPy3 2.1 beta 1 release here: http://pypy.org/download.html#pypy3-2-1-beta-1 Highlights ========== * The first release of PyPy3: support for Python 3, targetting CPython 3.2.3! - There are some `known issues`_ including performance regressions (issues `#1540`_ & `#1541`_) slated to be resolved before the final release. What is PyPy? ============== PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3 or 3.2.3. It's fast due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Also this release supports ARM machines running Linux 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` (like Beagleboard, Chromebook, Cubieboard, etc.) that supports ``VFPv3`` should work. Cheers, the PyPy team From naylor.b.david at gmail.com Wed Jul 31 21:20:59 2013 From: naylor.b.david at gmail.com (David Naylor) Date: Wed, 31 Jul 2013 22:20:59 +0300 Subject: [pypy-dev] Pypy-2.1-beta2 translation error Message-ID: <11961581.rsoAsemi6P@dragon.dg> Hi, While trying to self-translate pypy-2.1-beta2 I got the following error: RPython traceback: File "rpython_jit_metainterp_resume.c", line 7778, in blackhole_from_resumedata File "rpython_jit_metainterp_resume.c", line 9972, in ResumeDataDirectReader_consume_vref_and_vable File "rpython_jit_metainterp_resume.c", line 695, in ResumeDataDirectReader_consume_vable_info Fatal RPython error: AssertionError Is this a known problem or something wrong with my setup? Regards -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: This is a digitally signed message part. URL: From alex.gaynor at gmail.com Wed Jul 31 22:29:57 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 31 Jul 2013 13:29:57 -0700 Subject: [pypy-dev] Pypy-2.1-beta2 translation error In-Reply-To: <11961581.rsoAsemi6P@dragon.dg> References: <11961581.rsoAsemi6P@dragon.dg> Message-ID: What version of PyPy are you using to translate? Alex On Wed, Jul 31, 2013 at 12:20 PM, David Naylor wrote: > Hi, > > While trying to self-translate pypy-2.1-beta2 I got the following error: > > RPython traceback: > File "rpython_jit_metainterp_resume.c", line 7778, in > blackhole_from_resumedata > File "rpython_jit_metainterp_resume.c", line 9972, in > ResumeDataDirectReader_consume_vref_and_vable > File "rpython_jit_metainterp_resume.c", line 695, in > ResumeDataDirectReader_consume_vable_info > Fatal RPython error: AssertionError > > Is this a known problem or something wrong with my setup? > > Regards > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From naylor.b.david at gmail.com Wed Jul 31 22:57:22 2013 From: naylor.b.david at gmail.com (David Naylor) Date: Wed, 31 Jul 2013 22:57:22 +0200 Subject: [pypy-dev] Pypy-2.1-beta2 translation error In-Reply-To: References: <11961581.rsoAsemi6P@dragon.dg> Message-ID: On 31 Jul 2013 10:29 PM, "Alex Gaynor" wrote: > > What version of PyPy are you using to translate? Pypy 2.1-beta2 gives me this error. I had previously translated with beta1 but without this error (I can confirm this if needed). > On Wed, Jul 31, 2013 at 12:20 PM, David Naylor wrote: >> >> Hi, >> >> While trying to self-translate pypy-2.1-beta2 I got the following error: >> >> RPython traceback: >> File "rpython_jit_metainterp_resume.c", line 7778, in >> blackhole_from_resumedata >> File "rpython_jit_metainterp_resume.c", line 9972, in >> ResumeDataDirectReader_consume_vref_and_vable >> File "rpython_jit_metainterp_resume.c", line 695, in >> ResumeDataDirectReader_consume_vable_info >> Fatal RPython error: AssertionError >> >> Is this a known problem or something wrong with my setup? >> >> Regards >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev >> > > > > -- > "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: