From arigo at tunes.org Sat Jan 1 18:24:26 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 1 Jan 2011 18:24:26 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: Hi Paolo, Although I'm sure that your post is very interesting with the right mindset, I have a problem understanding the connection with Python. I know that it's exactly what you are complaining about, i.e. that Python developers appear not to care much about this deep Java/C++ issue. However, if you mean to do anything about it, you need to first understand yourself why it is so -- and the following reason is by far not enough: > As a matter of fact, this work is unsurprisingly often totally ignored > in many discussions about Python's memory model. I'm not surprised > because it's complex stuff, and understanding it doesn't matter for > today's Python programming. The most obvious misunderstanding, in my opinion, is that in Java or C++ it's about fields reads and writes, whereas in Python it's about any operation on any built-in type -- which can be for example reading or writing attributes, or insert()ing in a list, or doing setdefault() on a dictionary, or... any complex operation. This means that you cannot *at all* reduce the problem to field reads and field writes. As long as the discussion at the level of Java or C++ is only about fields, it's going to be ignored as "mostly uninteresting" by the Python folks. On the other hand, the links you posted about nonblockinghashmap are indeed interesting in this context. From a Python point of view, what is left to discuss is whether these hash maps offer enough "consistency" behavior to be usable on Python's default memory model. Or maybe it's not that interesting any more now that PyPy tends not to use a lot of dictionaries any more (the attributes of objects are not in a hash map, internally). I am myself just expressing vague, uninformed opinions again. But the final point is that this really needs someone motivated to experiment with PyPy (or CPython), and as long as no-one shows up, it will mostly be just "moving air around", i.e. wasting time to discuss this too much in depth. A bient?t, Armin. From tobami at googlemail.com Sun Jan 2 11:24:03 2011 From: tobami at googlemail.com (Miquel Torres) Date: Sun, 2 Jan 2011 11:24:03 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Hi all, I had a look at the issue and could finally pin it (more or less). Codespeed does destroy and recreate the whole element containing the canvas, so it is NOT a but in jqPlot (nor in Codespeed). It is a tricky GC issue where memory taken by big canvases is not properly freed. It probably affects several bowsers with modern JS JITs. Actually, in Firefox 3.6 it is so bad that not even closing the tab frees up the memory for a while! Chrome exhibits similar behaviour, but it does free old canvases after 20-40s. I am sure you will be interested in the fact that Konqueror 4.4 (with KHTML and kjs), which probably doesn't have such an aggressive JIT (or no JIT at all?), does NOT suffer this problem. Every refresh properly frees up the memory of the old canvas. For anyone further interested, Dmtrii did an excellent job filling a new bug for it with simple example code that triggers the problem (I added my observations there): https://bugzilla.mozilla.org/show_bug.cgi?id=621416 So what about speed.pypy.org? It being a multi-browser GC bug, I did the only thing I could do: limit the size of the sometimes ridiculously big horizontal plots ;-) It now has a 2000px limit. It does not completely fix the issue, but it is much more bearable. In the first 2 refreshes (selecting or -deselecting benchmarks), FF even manages to free up the memory. After that it keeps on eating more, but its much better. Thanks Dmitrii for caring! :-D (Please tell me whether it got better for you.) Cheers, Miquel 2010/12/23 Armin Rigo : > Hi Paolo, > > On Thu, Dec 23, 2010 at 3:09 PM, Paolo Giarrusso wrote: >> From your description, I guess that what you describe as "raw memory" >> is not accounted in the stats triggering GC. That would explain the >> behavior you describe, and suggest an easy fix. Indeed, one could wrap >> the raw-memory allocator to create and update such stats; then the >> heap-overflow check could consider them to decide whether to trigger >> GC. The details of the modified heap-overflow check would probably not >> be entirely trivial, but still doable. > > Absolutely. ?I will write it down in extradoc/planning/ for now. > > > Armin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From no-reply at dropboxmail.com Mon Jan 3 23:05:05 2011 From: no-reply at dropboxmail.com (Dropbox) Date: Mon, 03 Jan 2011 22:05:05 +0000 Subject: [pypy-dev] Wing Sit invited you to Dropbox Message-ID: <20110103220505.DB30E3F550D@mailman-2.dropboxmail.com> Wing Sit wants you to use Dropbox to sync and share files online and across computers. Get started here: http://www.dropbox.com/link/20.eHuq1oJQCM/NjU2NTA1ODEwNw?src=referrals_bulk2 - The Dropbox Team ____________________________________________________ To stop receiving invites from Dropbox, please go to http://www.dropbox.com/bl/4b1a4ffd25d4/pypy-dev%40codespeak.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmalcolm at redhat.com Mon Jan 3 23:13:38 2011 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 03 Jan 2011 17:13:38 -0500 Subject: [pypy-dev] PyPy is now available in Fedora Message-ID: <1294092818.26496.64.camel@radiator.bos.redhat.com> I've packaged pypy in RPM form for the Fedora distribution [1] - RPM packages are now built in the development branch targeting the next major release (Fedora 15). So it should now be possible for Fedora users to type # yum install pypy and obtain a precompiled /usr/bin/pypy executable via an rpm package, consisting of the JIT-enabled pypy, with all standard settings, without needing to do the full build themselves. I'll do my best to keep these "downstream" packages promptly updated as further "upstream" releases of PyPy occur. Many thanks to everyone who helped with this, especially fijal, for all his great feedback on the packaging review [2] (Caveat: actually, the build won't be available yet via "yum" until a cronjob runs tonight; for now, the build can be downloaded from: http://koji.fedoraproject.org/koji/buildinfo?buildID=212473 ) Some other links, for the curious, can be seen here: https://admin.fedoraproject.org/pkgdb/acls/name/pypy e.g. to the rpm packaging sources. Hope this is helpful Dave [1] http://fedoraproject.org/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=588941 From fijall at gmail.com Tue Jan 4 08:48:09 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 4 Jan 2011 09:48:09 +0200 Subject: [pypy-dev] PyPy is now available in Fedora In-Reply-To: <1294092818.26496.64.camel@radiator.bos.redhat.com> References: <1294092818.26496.64.camel@radiator.bos.redhat.com> Message-ID: Good job! On Tue, Jan 4, 2011 at 12:13 AM, David Malcolm wrote: > I've packaged pypy in RPM form for the Fedora distribution [1] - RPM > packages are now built in the development branch targeting the next > major release (Fedora 15). > > So it should now be possible for Fedora users to type > ?# yum install pypy > and obtain a precompiled > ?/usr/bin/pypy > executable via an rpm package, consisting of the JIT-enabled pypy, with > all standard settings, without needing to do the full build themselves. > > I'll do my best to keep these "downstream" packages promptly updated as > further "upstream" releases of PyPy occur. > > Many thanks to everyone who helped with this, especially fijal, for all > his great feedback on the packaging review [2] > > > (Caveat: actually, the build won't be available yet via "yum" until a > cronjob runs tonight; for now, the build can be downloaded from: > ?http://koji.fedoraproject.org/koji/buildinfo?buildID=212473 ) > > Some other links, for the curious, can be seen here: > ?https://admin.fedoraproject.org/pkgdb/acls/name/pypy > e.g. to the rpm packaging sources. > > Hope this is helpful > Dave > [1] http://fedoraproject.org/ > [2] https://bugzilla.redhat.com/show_bug.cgi?id=588941 > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From Ben.Young at sungard.com Tue Jan 4 09:40:31 2011 From: Ben.Young at sungard.com (Ben.Young at sungard.com) Date: Tue, 4 Jan 2011 08:40:31 +0000 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: References: Message-ID: <01781CA2CC22B145B230504679ECF48C02AA1FF7@EMEA-EXCHANGE03.internal.sungard.corp> > -----Original Message----- > From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- > bounces at codespeak.net] On Behalf Of Paolo Giarrusso > Sent: 24 December 2010 11:39 > To: Dima Tisnek > Cc: PyPy Dev; Armin Rigo > Subject: Re: [pypy-dev] pypy GC on large objects Re: > funding/popularity? > > On Thu, Dec 23, 2010 at 20:30, Dima Tisnek wrote: > > Basically collecting this is hard: > > > > dict(a=range(9**9)) > > > > large list is referenced, the object that holds the only reference is > > small no matter how you look at it. > First, usually (in most GC-ed languages) you can collect the list > before the dict. In PyPy, if finalizers are involved (is this the case > here? That'd be surprising), this is no more true. > > However, object size is not the point. For standard algorithms, the > size of an object does not matter at all in deciding when it's > collected - I already discussed this in my other email in this thread, > and I noted what actually could happen in the examples described by > Armin, and your examples show that it is a good property. A large > object in the same heap can fill it up and trigger an earlier garbage > collection. > > In general, if GC ran in the background (but it usually doesn't, and > not in PyPy) it could make sense to free objects sooner or later, > depending not on object size, but on "how much memory would be > 'indirectly freed' by freeing this object". However, because of > sharing, answering this question is too complex (it requires > collecting data from the whole heap). Moreover, the whole thing makes > no sense at all with usual, stop-the-world collectors: the app is > stopped, then the whole young generation, or the whole heap, is > collected, then the app is resumed. > > When separate heaps are involved (such as with ctypes, or with Large > Object Spaces, which avoid using a copy collector for large objects), > it is more complicated to ensure that the same property holds: you > need to consider stats of all heaps to decide whether to trigger GC. > > > I guess it gets harder still if there are many small live objects, as > > getting to this dict takes a while > > (easier in this simple case with generataional collector, O(n) in > general case) > > Not sure what you mean; I can make sense of it (not fully) only with > an incremental collector, and they are still used seldom (especially, > not in PyPy). > > Best regards > > > On 23 December 2010 06:38, Armin Rigo wrote: > >> Hi Ren?, > >> > >> On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield > wrote: > >>> I think this is a case where the object returned by > >>> ctypes.create_string_buffer() could use a correct __sizeof__ method > >>> return value. ?If pypy supported that, then the GC's could support > >>> extensions, and 'opaque' data structures in C too a little more > >>> nicely. > >> > >> I think you are confusing levels. ?There is no way the GC can call > >> some app-level Python method to get information about the objects it > >> frees (and when would it even call it?). ?Remember that our GC is > >> written at a level where it works for any interpreter for any > >> language, not just Python. > >> .NET supports calls to GC.AddMemoryPressure and GC.RemoveMemoryPressure to inform the GC you are allocating things outside of its knowledge. Maybe something similar would help? Cheers, Ben > >> > >> A bient?t, > >> > >> Armin. > >> _______________________________________________ > >> pypy-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/pypy-dev > >> > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From p.giarrusso at gmail.com Tue Jan 4 15:50:50 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Tue, 4 Jan 2011 15:50:50 +0100 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: <01781CA2CC22B145B230504679ECF48C02AA1FF7@EMEA-EXCHANGE03.internal.sungard.corp> References: <01781CA2CC22B145B230504679ECF48C02AA1FF7@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: On Tue, Jan 4, 2011 at 09:40, wrote: > > >> -----Original Message----- >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- >> bounces at codespeak.net] On Behalf Of Paolo Giarrusso >> Sent: 24 December 2010 11:39 >> To: Dima Tisnek >> Cc: PyPy Dev; Armin Rigo >> Subject: Re: [pypy-dev] pypy GC on large objects Re: >> funding/popularity? >> >> On Thu, Dec 23, 2010 at 20:30, Dima Tisnek wrote: >> > Basically collecting this is hard: >> > >> > dict(a=range(9**9)) >> > >> > large list is referenced, the object that holds the only reference is >> > small no matter how you look at it. >> First, usually (in most GC-ed languages) you can collect the list >> before the dict. In PyPy, if finalizers are involved (is this the case >> here? That'd be surprising), this is no more true. >> >> However, object size is not the point. For standard algorithms, the >> size of an object does not matter at all in deciding when it's >> collected - I already discussed this in my other email in this thread, >> and I noted what actually could happen in the examples described by >> Armin, and your examples show that it is a good property. A large >> object in the same heap can fill it up and trigger an earlier garbage >> collection. >> >> In general, if GC ran in the background (but it usually doesn't, and >> not in PyPy) it could make sense to free objects sooner or later, >> depending not on object size, but on "how much memory would be >> 'indirectly freed' by freeing this object". However, because of >> sharing, answering this question is too complex (it requires >> collecting data from the whole heap). Moreover, the whole thing makes >> no sense at all with usual, stop-the-world collectors: the app is >> stopped, then the whole young generation, or the whole heap, is >> collected, then the app is resumed. >> >> When separate heaps are involved (such as with ctypes, or with Large >> Object Spaces, which avoid using a copy collector for large objects), >> it is more complicated to ensure that the same property holds: you >> need to consider stats of all heaps to decide whether to trigger GC. >> >> > I guess it gets harder still if there are many small live objects, as >> > getting to this dict takes a while >> > (easier in this simple case with generataional collector, O(n) in >> general case) >> >> Not sure what you mean; I can make sense of it (not fully) only with >> an incremental collector, and they are still used seldom (especially, >> not in PyPy). >> >> Best regards >> >> > On 23 December 2010 06:38, Armin Rigo wrote: >> >> Hi Ren?, >> >> >> >> On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield >> wrote: >> >>> I think this is a case where the object returned by >> >>> ctypes.create_string_buffer() could use a correct __sizeof__ method >> >>> return value. ?If pypy supported that, then the GC's could support >> >>> extensions, and 'opaque' data structures in C too a little more >> >>> nicely. >> >> >> >> I think you are confusing levels. ?There is no way the GC can call >> >> some app-level Python method to get information about the objects it >> >> frees (and when would it even call it?). ?Remember that our GC is >> >> written at a level where it works for any interpreter for any >> >> language, not just Python. >> >> > > > .NET supports calls to GC.AddMemoryPressure and GC.RemoveMemoryPressure to inform the GC you are allocating things outside of its knowledge. Maybe something similar would help? That's interesting as well. I and Armin discussed something similar in another branch of this thread, and he included that among planned ideas: http://codespeak.net/pipermail/pypy-dev/2010q4/006648.html http://codespeak.net/pipermail/pypy-dev/2010q4/006649.html The difference is that in my proposal one would hook the memory allocator for Python extensions, the .NET requires adding explicit calls to the source code. However, the key idea is that you might need to GC sooner if there is lots of unmanaged memory. Unfortunately, MSDN docs about those methods do not give pointers to the heuristics used: http://msdn.microsoft.com/en-us/library/system.gc.addmemorypressure.aspx http://msdn.microsoft.com/en-us/library/system.gc.removememorypressure.aspx Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From duncan at grisby.org Tue Jan 4 18:25:36 2011 From: duncan at grisby.org (Duncan Grisby) Date: Tue, 04 Jan 2011 17:25:36 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ Message-ID: <1294161936.1893.20.camel@localhost> Hi, I've been experimenting with using PyPy in a big application, and the initial results are very encouraging. One part of the application is a fairly complex compiler, and that goes 2.5 times faster with PyPy than with Python 2.6, which is a very positive result. The compiler is the one CPU-intensive bit of the system that I could easily isolate to test with PyPy. All the rest of it consists of a number of servers connected together using omniORB (which I maintain). omniORBpy integrates the C++ core of omniORB with Python, via the C API. omniORB is multi-threaded, and needs to call into Python from threads created in the C++ world. With CPython that's possible, although it takes some care to behave properly with thread states and the interpreter lock and so on. As far as I can see, PyPy's C API implementation doesn't support the thread state functions like PyGILState_Ensure and PyThreadState_Get, which isn't a huge surprise given how different PyPy is. Are these functions likely to be implemented at any point? Alternatively, is there any other way for C/C++ code to call into PyPy from a thread created outside PyPy? If there isn't a mechanism to do it at the moment, is it possible at all? How much effort would it be to create one? Thanks, Duncan. -- -- Duncan Grisby -- -- duncan at grisby.org -- -- http://www.grisby.org -- From alex.gaynor at gmail.com Wed Jan 5 21:51:18 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 5 Jan 2011 14:51:18 -0600 Subject: [pypy-dev] psycopg2 on PyPy Message-ID: Hi all, For the past week and a half I've been working on an RPython implementation of psycopg2 (the most popular database adapter for PostgreSQL). I'm happy to announce that at this point it is passing all the DB-API2.0 tests, and all the psycopg2 tests I converted into AppTests (most of them I think), it also passes the entire DJango test suite (which makes extensive use of the database). The downside is that running the full Django test suite on top of pypy with it is still about 20% slower than CPython. So we have some work to do. The code is all at: http://bitbucket.org/alex_gaynor/pypy-postgresql, I don't plan on merging this back into the main dev tree, as it doesn't seem appropriate for pypy-core to have a PostgreSQL adapter, rather I'd like to spend some time looking at seperate compilation, so we can have external RPython modules. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: From santagada at gmail.com Thu Jan 6 03:29:12 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 6 Jan 2011 00:29:12 -0200 Subject: [pypy-dev] psycopg2 on PyPy In-Reply-To: References: Message-ID: On Wed, Jan 5, 2011 at 6:51 PM, Alex Gaynor wrote: > Hi all, > For the past week and a half I've been working on an RPython implementation > of psycopg2 (the most popular database adapter for PostgreSQL). ?I'm happy > to announce that at this point it is passing all the DB-API2.0 tests, and > all the psycopg2 tests I converted into AppTests (most of them I think), it > also passes the entire DJango test suite (which makes?extensive?use of the > database). ?The downside is that running the full Django test suite on top > of pypy with it is still about 20% slower than CPython. ?So we have some > work to do. ?The code is all > at:?http://bitbucket.org/alex_gaynor/pypy-postgresql, I don't plan on > merging this back into the main dev tree, as it doesn't seem appropriate for > pypy-core to have a PostgreSQL adapter, rather I'd like to spend some time > looking at seperate compilation, so we can have external RPython modules. > What do you think are the main performance problem? Is it just the nature of unittests that are too small for the jit to work, is it that you have maybe too much rpython code that is not being jitted as well as it could or is it just a performance problem of pypy? -- Leonardo Santagada From amauryfa at gmail.com Thu Jan 6 03:59:43 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 6 Jan 2011 03:59:43 +0100 Subject: [pypy-dev] psycopg2 on PyPy In-Reply-To: References: Message-ID: Hi, 2011/1/5 Alex Gaynor : > The code is all at:?http://bitbucket.org/alex_gaynor/pypy-postgresql, I > don't plan on merging this back into the main dev tree, as it doesn't seem > appropriate for pypy-core to have a PostgreSQL adapter, There is already a module/oracle directory in the pypy tree, so why not... but you should be careful with the license: If your work is a translation of psycopg2, it probably needs to be licensed under the GPL. so this would be incompatible with PyPy which is licensed under a MIT-like license. (the oracle module in pypy is based on cx_Oracle, which also uses the MIT license) > rather I'd like to > spend some time looking at seperate compilation, so we can have external > RPython modules. If your goal is to build RPython modules incrementally, after the pypy interpreter has been compiled, it it a very difficult task, and at least three attempts to do it have failed. (The fourth one eventually produced the cpyext module, which is not a complete failure, but has nothing to do with RPython) On the other hand, it would be handy to be able to translate modules which source is outside pypy tree; something like:: ./translate.py -Ojit targetpypystandalone.py --with-external-module=/path/to/my/psycopg2 -- Amaury Forgeot d'Arc From renesd at gmail.com Thu Jan 6 14:57:13 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 6 Jan 2011 13:57:13 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: <1294161936.1893.20.camel@localhost> References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, are the thread GIL releasing functions going to be supported in pypy with CPyExt? To allow this usage: PyThreadState *_save; _save = PyEval_SaveThread(); // ...Do some blocking I/O operation, or CPU intensive operation that does not use the python api... PyEval_RestoreThread(_save); Or using the macros... Py_BEGIN_ALLOW_THREADS; // ...Do some blocking I/O operation, or CPU intensive operation that does not use the python api... Py_END_ALLOW_THREADS; Is it not possible or too hard, or is it just not implemented yet? On Tue, Jan 4, 2011 at 5:25 PM, Duncan Grisby wrote: > Hi, > > I've been experimenting with using PyPy in a big application, and the > initial results are very encouraging. One part of the application is a > fairly complex compiler, and that goes 2.5 times faster with PyPy than > with Python 2.6, which is a very positive result. > > The compiler is the one CPU-intensive bit of the system that I could > easily isolate to test with PyPy. All the rest of it consists of a > number of servers connected together using omniORB (which I maintain). > omniORBpy integrates the C++ core of omniORB with Python, via the C API. > omniORB is multi-threaded, and needs to call into Python from threads > created in the C++ world. With CPython that's possible, although it > takes some care to behave properly with thread states and the > interpreter lock and so on. > > As far as I can see, PyPy's C API implementation doesn't support the > thread state functions like PyGILState_Ensure and PyThreadState_Get, > which isn't a huge surprise given how different PyPy is. Are these > functions likely to be implemented at any point? ?Alternatively, is > there any other way for C/C++ code to call into PyPy from a thread > created outside PyPy? > > If there isn't a mechanism to do it at the moment, is it possible at > all? ?How much effort would it be to create one? > > Thanks, > > Duncan. > > -- > ?-- Duncan Grisby ? ? ? ? -- > ?-- duncan at grisby.org ? ? -- > ? -- http://www.grisby.org -- > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From amauryfa at gmail.com Thu Jan 6 15:13:39 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 6 Jan 2011 15:13:39 +0100 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, 2011/1/6 Ren? Dudfield : > Hi, > > are the thread GIL releasing functions going to be supported in pypy > with CPyExt? > > To allow this usage: > > PyThreadState *_save; > _save = PyEval_SaveThread(); > // ...Do some blocking I/O operation, or CPU intensive operation that > does not use the python api... > PyEval_RestoreThread(_save); > > Or using the macros... > > Py_BEGIN_ALLOW_THREADS; > // ...Do some blocking I/O operation, or CPU intensive operation that > does not use the python api... > Py_END_ALLOW_THREADS; > > Is it not possible or too hard, or is it just not implemented yet? Frankly, I even don't know. At least, the Py_BEGIN_ALLOW_THREADS macros are there and will compile. OTOH it seems that every call from the pypy interpreter to C releases the GIL, and that conversely, every call from to a Python API function will grab the GIL for the duration of the call (yes, even Py_INCREF) cpyext is maybe already thread-safe, only awfully slow... But this is just a guess, from looking at the source code. I'd be grateful if you could write a custom module and do some tests in this area, with and without the Py_BEGIN_ALLOW_THREADS calls. Cheers, -- Amaury Forgeot d'Arc From renesd at gmail.com Thu Jan 6 15:35:46 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 6 Jan 2011 14:35:46 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, here is a basic function... without the whole module stuff around it. I'd need to do the rest of the module another time. PyObject* do_long_run (PyObject* self) { int i, a, b, c; a = 2; Py_BEGIN_ALLOW_THREADS; for(i = 0; i < 2000000000; i++) { b = i + a; c = a + i; } a = c - b; Py_END_ALLOW_THREADS; Py_RETURN_NONE; } // ... { "do_long_run", (PyCFunction) do_long_run , METH_NOARGS, "runs some cpu intensive code whilst releasing the GIL"}, // ... You could test it by calling that function in separate threads. With another thread printing something from python... like in the python code below. from threading import Thread # this assumes the _long_runner is the C module with the do_long_run function in it. import _long_runner class PythonPrinter(Thread): def run(self): for x in range(1000000): print (x) class CLongRunner(Thread): def run(self): _long_runner.do_long_run() threads = [PythonPrinter(), CLongRunner()] for t in threads: t.start() for t in threads: t.join() That is untested, and typed without compiling/running... but it should work, and should give you some insight on how the threading works. Maybe you can throw the c function into an existing module for easier testing... otherwise maybe I can write it in a few days. cya. On Thu, Jan 6, 2011 at 2:13 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/1/6 Ren? Dudfield : >> Hi, >> >> are the thread GIL releasing functions going to be supported in pypy >> with CPyExt? >> >> To allow this usage: >> >> PyThreadState *_save; >> _save = PyEval_SaveThread(); >> // ...Do some blocking I/O operation, or CPU intensive operation that >> does not use the python api... >> PyEval_RestoreThread(_save); >> >> Or using the macros... >> >> Py_BEGIN_ALLOW_THREADS; >> // ...Do some blocking I/O operation, or CPU intensive operation that >> does not use the python api... >> Py_END_ALLOW_THREADS; >> >> Is it not possible or too hard, or is it just not implemented yet? > > Frankly, I even don't know. > At least, the Py_BEGIN_ALLOW_THREADS macros are there and will compile. > > OTOH it seems that every call from the pypy interpreter to C releases the GIL, > and that conversely, every call from to a Python API function will > grab the GIL for the duration of the call (yes, even Py_INCREF) > cpyext is maybe already thread-safe, only awfully slow... > But this is just a guess, from looking at the source code. > > I'd be grateful if you could write a custom module and do some tests > in this area, > with and without the Py_BEGIN_ALLOW_THREADS calls. > > Cheers, > > -- > Amaury Forgeot d'Arc > From nathanael.jones at gmail.com Mon Jan 10 05:24:09 2011 From: nathanael.jones at gmail.com (Nathanael D. Jones) Date: Sun, 9 Jan 2011 23:24:09 -0500 Subject: [pypy-dev] Continuations and sandboxing Message-ID: Hi folks, I've got a challenging set of requirements that I think PyPy may be able to meet, but I'd like to get some feedback before I jump in. I'm developing a collaboratively edited game. It's not a MUD, but similar in concept, and also text based in nature. HogwartsLive.com and lotgd.ne t are examples of the genre I'm going for. I believe if a game of that genre makes user modification simple and rewarding enough, it could be self-sustaining and grow quickly enough to keep users interested perpetually, like Wikipedia. It's also a fascinating computer science challenge, because it requires a LOT from a computer language. 1) Sandboxing. To allow users to make changes to the game without being able to "cheat" at it or escalate privileges. 2) Serializeable continuations. With gameplay being based on good plot and story flow, continuations are critical to allow straightforward implementation of 'workflows' that depend on user choice at every turn. 3) Tail-call elimination. By nature, players will accumulate a very large call stack. While this isn't terribly bad a first glance, the following issue combines with it to cause a very big problem: When code changes underneath a continuation, we need to determine how to resume flow. One option is annotating a checkpoint method in each code 'file'. However, if a user's call stack includes every file in the system, each change will cause them to restart. Tail-call elimination would help eliminate unneeded stack frames and minimize re-spawning. 3) Dynamic, strong typing and metaprogramming are key for keeping the API simple. 4) Dynamic code loading. Users will be able to 'branch' their own version of the world and share it with others. There may be thousands of versions of a class, and they need to be able to execute in separate sandboxes at the same time. Source code will be pulled from a Git repository or some kind of versioning database. I'm interested in knowing which of these PyPy already does, and which of them I can make it do. I appreciate your help! Nathanael Jones http://nathanaeljones.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dimaqq at gmail.com Mon Jan 10 06:24:22 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Sun, 9 Jan 2011 22:24:22 -0700 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: First of all, congratulations on getting THE idea, I think user content is essential too, but I see why commercial games shy away from it An open source game could do really well! 1, yes, but not in a way you want (you want many boxes that can't touch each other, right?) 2, yes, stackless is the name/misnomer 3a, I doubt is relevant, you have to show that your workload has a substantial number of tail calls; stack depth is not that important with stackless. 3b, I imagine this is impossible in general case, how do you see this done in any language? 3 again, yes of course, and I bet you knew this already. 4 indeed, unloading old code is a bit harder as you have to make sure all references to module or function or class are lost in timely fashion. p.s. I'm not an expert on pypy yet On 9 January 2011 21:24, Nathanael D. Jones wrote: > Hi folks, > I've got a challenging set of requirements that I think PyPy may be able to > meet, but I'd like to get some feedback before I jump in. > I'm developing a collaboratively?edited?game. It's not a MUD, but?similar?in > concept, and also text based in nature. > HogwartsLive.com and?lotgd.net?are examples of the genre I'm going for. > I believe if a game of that genre makes user modification simple and > rewarding enough, it could be self-sustaining?and grow quickly enough to > keep users interested perpetually, like Wikipedia. > It's also a fascinating computer science challenge, because it requires a > LOT from a computer language. > 1) Sandboxing. To allow users to make changes to the game without being able > to "cheat" at it or escalate?privileges. > 2) Serializeable continuations. With gameplay being based on good plot and > story flow, continuations are critical to allow straightforward > implementation of 'workflows' that depend on user choice at every turn. > 3) Tail-call elimination. ?By nature, players will accumulate a very large > call stack. While this isn't terribly bad a first glance, the following > issue combines with it to cause a very big problem: > When code changes underneath a continuation, we need to determine how to > resume flow. One option is annotating a checkpoint method in each code > 'file'. However, if a user's call stack includes every file in the system, > each change will cause them to restart. > Tail-call elimination would help eliminate unneeded stack frames and > minimize re-spawning. > 3) Dynamic, strong typing and metaprogramming are key for keeping the API > simple. > 4) Dynamic code loading. Users will be able to 'branch' their own version of > the world and share it with others. There may be thousands of versions of a > class, and they need to be able to execute in separate sandboxes at the same > time. Source code will be pulled from a Git repository or some kind of > versioning database. > I'm interested in knowing which of these PyPy already does, and which of > them I can make it do. > I appreciate your help! > Nathanael Jones > http://nathanaeljones.com/ > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From william.leslie.ttg at gmail.com Mon Jan 10 09:22:27 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Mon, 10 Jan 2011 19:22:27 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: On 10 January 2011 15:24, Nathanael D. Jones wrote: > Hi folks, Hi! > 2) Serializeable continuations. With gameplay being based on good plot and > story flow, continuations are critical to allow straightforward > implementation of 'workflows' that depend on user choice at every turn. > 3) Tail-call elimination. ?By nature, players will accumulate a very large > call stack. While this isn't terribly bad a first glance, the following > issue combines with it to cause a very big problem: > When code changes underneath a continuation, we need to determine how to > resume flow. One option is annotating a checkpoint method in each code > 'file'. However, if a user's call stack includes every file in the system, > each change will cause them to restart. > Tail-call elimination would help eliminate unneeded stack frames and > minimize re-spawning. I wonder if you don't want instead to manage the (user) dynamic context yourself - if players already need to remember to write all actions in a tail-recursive manner, it is probably worth making that API explicit, and if you do that, the underlying implementation doesn't need to do TCE at all. But I guess the existing stackless support should be good enough if you just want one-shot continuations. > 4) Dynamic code loading. Users will be able to 'branch' their own version of > the world and share it with others. There may be thousands of versions of a > class, and they need to be able to execute in separate sandboxes at the same > time. Source code will be pulled from a Git repository or some kind of > versioning database. Quite like this idea. You do have to deal with a bunch of (fairly well known) problems, which any specific implementation of dynamic code loading is going to need to solve (or not). Pypy doesn't currently implement any hot-schema-change magic, and reloading has always been error prone in the presence of state. First-class mutable types make it particularly difficult (there is no sensible answer to what it means to reload a python class). The one issue that interests me is where you implement the persistence boundary - do you go with orthogonal persistence and act as if everything is preserved, or assume all user code is run within some sort of (fast and loose) transaction that can be re-entered at will, providing an API for persistent data access? The second case makes the reloading question a bit more reasonable, because you can always throw away the current delta and replay the external effects, assuming the interface for the external events hasn't changed significantly. -- William Leslie From fijall at gmail.com Mon Jan 10 12:23:12 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 10 Jan 2011 13:23:12 +0200 Subject: [pypy-dev] [pypy-svn] pypy jit-longlong: Intermediate check-in, breaks everything by exposing cases of In-Reply-To: <20110109223334.6A6D9282B9E@codespeak.net> References: <20110109223334.6A6D9282B9E@codespeak.net> Message-ID: For what is worth I'm happy with making "rething regalloc API" a sprint topic. So far I failed to come up with something more reasonable. Cheers, fijal On Mon, Jan 10, 2011 at 12:33 AM, arigo wrote: > Author: Armin Rigo > Branch: jit-longlong > Changeset: r40545:9a472ddf9893 > Date: 2011-01-09 23:33 +0100 > http://bitbucket.org/pypy/pypy/changeset/9a472ddf9893/ > > Log: ? ?Intermediate check-in, breaks everything by exposing cases of > ? ? ? ?constant arguments that are not correctly handled so far. /me is > ? ? ? ?again annoyed by the API of regalloc.py, which seems to be very > ? ? ? ?reasonable, until we hit these cases. > > diff --git a/pypy/jit/backend/x86/assembler.py b/pypy/jit/backend/x86/assembler.py > --- a/pypy/jit/backend/x86/assembler.py > +++ b/pypy/jit/backend/x86/assembler.py > @@ -1174,6 +1174,13 @@ > ? ? ? ? self.mc.SBB_rr(resloc.value, resloc.value) > ? ? ? ? self.mc.NEG_r(resloc.value) > > + ? ?def genop_llong_lt(self, op, arglocs, resloc): > + ? ? ? ?# XXX just a special case for now: "x < 0" > + ? ? ? ?loc1, = arglocs > + ? ? ? ?self.mc.PMOVMSKB_rx(resloc.value, loc1.value) > + ? ? ? ?self.mc.SHR_ri(resloc.value, 7) > + ? ? ? ?self.mc.AND_ri(resloc.value, 1) > + > ? ? def genop_new_with_vtable(self, op, arglocs, result_loc): > ? ? ? ? assert result_loc is eax > ? ? ? ? loc_vtable = arglocs[-1] > @@ -1784,6 +1791,7 @@ > ? ? ? ? ? ? if isinstance(op.getdescr(), LongLongCallDescr): > ? ? ? ? ? ? ? ? self.mc.MOV_br(resloc.value, eax.value) ? ? ?# long long > ? ? ? ? ? ? ? ? self.mc.MOV_br(resloc.value + 4, edx.value) > + ? ? ? ? ? ? ? ?# XXX should ideally not move the result on the stack > ? ? ? ? ? ? else: > ? ? ? ? ? ? ? ? self.mc.FSTP_b(resloc.value) ? # float return > ? ? ? ? elif size == WORD: > > diff --git a/pypy/jit/codewriter/jtransform.py b/pypy/jit/codewriter/jtransform.py > --- a/pypy/jit/codewriter/jtransform.py > +++ b/pypy/jit/codewriter/jtransform.py > @@ -894,7 +894,8 @@ > ? ? ? ? ? ? ? ? args = op.args > ? ? ? ? ? ? ? ? op1 = self.prepare_builtin_call(op, "llong_%s", args) > ? ? ? ? ? ? ? ? op2 = self._handle_oopspec_call(op1, args, > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EffectInfo.OS_LLONG_%s) > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EffectInfo.OS_LLONG_%s, > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EffectInfo.EF_PURE) > ? ? ? ? ? ? ? ? return op2 > ? ? ? ? ''' % (_op, _oopspec.lower(), _oopspec)).compile() > > @@ -1273,7 +1274,7 @@ > > ? ? def _handle_oopspec_call(self, op, args, oopspecindex, extraeffect=None): > ? ? ? ? calldescr = self.callcontrol.getcalldescr(op, oopspecindex) > - ? ? ? ?if extraeffect: > + ? ? ? ?if extraeffect is not None: > ? ? ? ? ? ? calldescr.get_extra_info().extraeffect = extraeffect > ? ? ? ? if isinstance(op.args[0].value, str): > ? ? ? ? ? ? pass ?# for tests only > > diff --git a/pypy/jit/backend/x86/regalloc.py b/pypy/jit/backend/x86/regalloc.py > --- a/pypy/jit/backend/x86/regalloc.py > +++ b/pypy/jit/backend/x86/regalloc.py > @@ -653,6 +653,22 @@ > ? ? ? ? self.PerformLLong(op, [loc1, loc2, loc3], loc0) > ? ? ? ? self.xrm.possibly_free_vars(args) > > + ? ?def _maybe_consider_llong_lt(self, op): > + ? ? ? ?# XXX just a special case for now > + ? ? ? ?from pypy.rlib.longlong2float import longlong2float > + ? ? ? ?box = op.getarg(2) > + ? ? ? ?if not isinstance(box, ConstFloat): > + ? ? ? ? ? ?return False > + ? ? ? ?if not (box.value == longlong2float(r_longlong(0))): > + ? ? ? ? ? ?return False > + ? ? ? ?# "x < 0" > + ? ? ? ?box = op.getarg(1) > + ? ? ? ?loc1 = self.xrm.make_sure_var_in_reg(box, imm_fine=False) > + ? ? ? ?loc0 = self.rm.force_allocate_reg(op.result) > + ? ? ? ?self.PerformLLong(op, [loc1], loc0) > + ? ? ? ?self.xrm.possibly_free_var(box) > + ? ? ? ?return True > + > ? ? def _consider_llong_to_int(self, op): > ? ? ? ? # accept an argument in a xmm register or in the stack > ? ? ? ? loc1 = self.xrm.loc(op.getarg(1)) > @@ -769,6 +785,9 @@ > ? ? ? ? ? ? ? ? if (oopspecindex == EffectInfo.OS_LLONG_EQ or > ? ? ? ? ? ? ? ? ? ? oopspecindex == EffectInfo.OS_LLONG_NE): > ? ? ? ? ? ? ? ? ? ? return self._consider_llong_cmp_xx(op) > + ? ? ? ? ? ? ? ?if oopspecindex == EffectInfo.OS_LLONG_LT: > + ? ? ? ? ? ? ? ? ? ?if self._maybe_consider_llong_lt(op): > + ? ? ? ? ? ? ? ? ? ? ? ?return > ? ? ? ? # > ? ? ? ? self._consider_call(op) > > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From p.giarrusso at gmail.com Mon Jan 10 21:18:05 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 10 Jan 2011 21:18:05 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: Hi all, On Mon, Jan 10, 2011 at 09:22, William ML Leslie wrote: > On 10 January 2011 15:24, Nathanael D. Jones wrote: >> Hi folks, >> 2) Serializeable continuations. With gameplay being based on good plot and >> story flow, continuations are critical to allow straightforward >> implementation of 'workflows' that depend on user choice at every turn. >> 3) Tail-call elimination. ?By nature, players will accumulate a very large >> call stack. While this isn't terribly bad a first glance, the following >> issue combines with it to cause a very big problem: >> When code changes underneath a continuation, we need to determine how to >> resume flow. One option is annotating a checkpoint method in each code >> 'file'. [I didn't get this part. Reading what's below, I assume that for each call frame, you remember somehow the file defining the called function.] > > However, if a user's call stack includes every file in the system, >> each change will cause them to restart. >> Tail-call elimination would help eliminate unneeded stack frames and >> minimize re-spawning. That optimization looks maybe invalid. Suppose file A contains a function f() which tail-calls g(1, 2, 3), and then file A is modified, so that f() does another tail-call. Now it's not clear why do you do restart: if you do restart when executed code is modified, then this optimization is invalid. If you reload only when code yet to execute is modified, then the optimization is valid, but you could perform also more advanced optimizations to avoid restart (you could compare the generated bytecode up to the end of the outermost loop containing the point where the call frame was saved and another procedure was invoked). It is also not clear which semantic guarantee you want to achieve by this restart. Would you use transactions to avoid performing side-effects again? >> 4) Dynamic code loading. Users will be able to 'branch' their own version of >> the world and share it with others. There may be thousands of versions of a >> class, and they need to be able to execute in separate sandboxes at the same >> time. Source code will be pulled from a Git repository or some kind of >> versioning database. > Quite like this idea. > You do have to deal with a bunch of (fairly well known) problems, > which any specific implementation of dynamic code loading is going to > need to solve (or not). ?Pypy doesn't currently implement any > hot-schema-change magic, and reloading has always been error prone in > the presence of state. ?First-class mutable types make it particularly > difficult (there is no sensible answer to what it means to reload a > python class). You might want to reuse the solutions to those issues used in the Java (and maybe .NET) world. Java allows reloading a class in a different classloader, and that has been used inside OSGi (see http://www.osgi.org/About/Technology). Not sure about the solution in OSGi, but Java serialization allows to serialize an instance of version 1 of a class, and to de-serialize it with version 2 of that class, if v2 takes extra care for that; one could use this to convert existing instances. > The one issue that interests me is where you implement the persistence > boundary - do you go with orthogonal persistence and act as if > everything is preserved, or assume all user code is run within some > sort of (fast and loose) transaction that can be re-entered at will, > providing an API for persistent data access? ?The second case makes > the reloading question a bit more reasonable, because you can always > throw away the current delta and replay the external effects, assuming > the interface for the external events hasn't changed significantly. The key question is: when would you start and commit such transactions? Apart from that, your idea looks very similar to Software Transactional Memory (STM). STM restarts explicitly-marked transactions in a thread when some other thread modifies the affected data (which would be a data race) and commits its transaction. In your case, a transaction is restarted when some other thread modifies the involved code. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From william.leslie.ttg at gmail.com Mon Jan 10 23:05:36 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 11 Jan 2011 09:05:36 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: On 11 January 2011 07:18, Paolo Giarrusso wrote: > Hi all, > > On Mon, Jan 10, 2011 at 09:22, William ML Leslie > wrote: >> On 10 January 2011 15:24, Nathanael D. Jones wrote: >>> 4) Dynamic code loading. Users will be able to 'branch' their own version of >>> the world and share it with others. There may be thousands of versions of a >>> class, and they need to be able to execute in separate sandboxes at the same >>> time. Source code will be pulled from a Git repository or some kind of >>> versioning database. > >> Quite like this idea. > >> You do have to deal with a bunch of (fairly well known) problems, >> which any specific implementation of dynamic code loading is going to >> need to solve (or not). ?Pypy doesn't currently implement any >> hot-schema-change magic, and reloading has always been error prone in >> the presence of state. ?First-class mutable types make it particularly >> difficult (there is no sensible answer to what it means to reload a >> python class). > > You might want to reuse the solutions to those issues used in the Java > (and maybe .NET) world. Java allows reloading a class in a different > classloader, and that has been used inside OSGi (see > http://www.osgi.org/About/Technology). > Not sure about the solution in OSGi, but Java serialization allows to > serialize an instance of version 1 of a class, and to de-serialize it > with version 2 of that class, if v2 takes extra care for that; one > could use this to convert existing instances. Sure, and there is also Dynamic Class Evolution for Hotspot, and many other VMs support some form of reloading and redefinition (goops is an exceptional and accessible example, see http://wingolog.org/archives/2009/11/09/class-redefinition-in-guile for a post with lots of diagrams). What I am trying to say is that the *ability* to do reloading does not mean your code will work in the presence of interface changes. You have to decide whether updating a closure in a way that captures more variables or changes the signature updates existing instances of that closure: if it does, there will be entries missing in the closure slot, if it doesn't, older closures won't be callable in the same way as the new closures. No matter what, you here need to write code to be explicitly reloadable, or provide some way to instruct the runtime what to do for each schema change. > >> The one issue that interests me is where you implement the persistence >> boundary - do you go with orthogonal persistence and act as if >> everything is preserved, or assume all user code is run within some >> sort of (fast and loose) transaction that can be re-entered at will, >> providing an API for persistent data access? ?The second case makes >> the reloading question a bit more reasonable, because you can always >> throw away the current delta and replay the external effects, assuming >> the interface for the external events hasn't changed significantly. > > The key question is: when would you start and commit such transactions? > > Apart from that, your idea looks very similar to Software > Transactional Memory (STM). STM restarts explicitly-marked > transactions in a thread when some other thread modifies the affected > data (which would be a data race) and commits its transaction. In your > case, a transaction is restarted when some other thread modifies the > involved code. There is a bit of ambiguity in the "some other thread modifies" here. I don't know what synchronisation and communication is going on in your game, but I suspect that it only rarely interacts with reloading code in an interesting way. I'll reply to this properly in another email, I'd better get back to work :) -- William Leslie From nathanael.jones at gmail.com Tue Jan 11 05:25:04 2011 From: nathanael.jones at gmail.com (Nathanael D. Jones) Date: Mon, 10 Jan 2011 23:25:04 -0500 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: Regardng #1: Sandboxing is a major concern for me. Different code will need different sandboxing levels depending upon who created/approved the code. I can't have everything in one sandbox - I need isolated boxes on a per-request level. I think I see a way to sidestep the need for #3 and also help with hot-swapping. If I try to empty the stack as often as possible, it should provide frequent opportunities for 'free' reloading. I.E. (psuedocode) def garden(): if btnHouse.clicked: return house def house(): if btnGarden.clicked: return garden def loop(): func = house; while(true): #TODO: Update func to the latest version #TODO: Build an execution sandbox appropriate to the security clearance func has func = sandbox.call(func) print "Your are now in the " + func The idea is that if functions return the next function to call instead of calling them, we have explicit tail-call elimination, and we have an explicit point at which we can rebuild the sandbox and upgrade the code. Regarding the persistence boundary, I've seen some very good points come up. A certain amount of orthogonal persistence is needed in the form of a continuation, but that would only be function-scoped data. I don't intent to allow users to use global variables or suchlike to persist data, I want a tailored API. Much data will be user scoped, and therefore lockable. However, some data will be shared across all users. I'm not sure what the best way to handle this is. With sandboxing and full control over the persistence API I could theoretically implement STM. I want to make sure the architecture is very scalable, so I've been considering something like BigTable or SimpleDB as the persistence store. Here transaction and locking options are more limited. Thoughts? Nathanael On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie < william.leslie.ttg at gmail.com> wrote: > On 11 January 2011 07:18, Paolo Giarrusso wrote: > > Hi all, > > > > On Mon, Jan 10, 2011 at 09:22, William ML Leslie > > wrote: > >> On 10 January 2011 15:24, Nathanael D. Jones > wrote: > >>> 4) Dynamic code loading. Users will be able to 'branch' their own > version of > >>> the world and share it with others. There may be thousands of versions > of a > >>> class, and they need to be able to execute in separate sandboxes at the > same > >>> time. Source code will be pulled from a Git repository or some kind of > >>> versioning database. > > > >> Quite like this idea. > > > >> You do have to deal with a bunch of (fairly well known) problems, > >> which any specific implementation of dynamic code loading is going to > >> need to solve (or not). Pypy doesn't currently implement any > >> hot-schema-change magic, and reloading has always been error prone in > >> the presence of state. First-class mutable types make it particularly > >> difficult (there is no sensible answer to what it means to reload a > >> python class). > > > > You might want to reuse the solutions to those issues used in the Java > > (and maybe .NET) world. Java allows reloading a class in a different > > classloader, and that has been used inside OSGi (see > > http://www.osgi.org/About/Technology). > > Not sure about the solution in OSGi, but Java serialization allows to > > serialize an instance of version 1 of a class, and to de-serialize it > > with version 2 of that class, if v2 takes extra care for that; one > > could use this to convert existing instances. > > Sure, and there is also Dynamic Class Evolution for Hotspot, and many > other VMs support some form of reloading and redefinition (goops is an > exceptional and accessible example, see > http://wingolog.org/archives/2009/11/09/class-redefinition-in-guile > for a post with lots of diagrams). > > What I am trying to say is that the *ability* to do reloading does not > mean your code will work in the presence of interface changes. You > have to decide whether updating a closure in a way that captures more > variables or changes the signature updates existing instances of that > closure: if it does, there will be entries missing in the closure > slot, if it doesn't, older closures won't be callable in the same way > as the new closures. No matter what, you here need to write code to > be explicitly reloadable, or provide some way to instruct the runtime > what to do for each schema change. > > > > >> The one issue that interests me is where you implement the persistence > >> boundary - do you go with orthogonal persistence and act as if > >> everything is preserved, or assume all user code is run within some > >> sort of (fast and loose) transaction that can be re-entered at will, > >> providing an API for persistent data access? The second case makes > >> the reloading question a bit more reasonable, because you can always > >> throw away the current delta and replay the external effects, assuming > >> the interface for the external events hasn't changed significantly. > > > > The key question is: when would you start and commit such transactions? > > > > Apart from that, your idea looks very similar to Software > > Transactional Memory (STM). STM restarts explicitly-marked > > transactions in a thread when some other thread modifies the affected > > data (which would be a data race) and commits its transaction. In your > > case, a transaction is restarted when some other thread modifies the > > involved code. > > There is a bit of ambiguity in the "some other thread modifies" here. > I don't know what synchronisation and communication is going on in > your game, but I suspect that it only rarely interacts with reloading > code in an interesting way. I'll reply to this properly in another > email, I'd better get back to work :) > > -- > William Leslie > -------------- next part -------------- An HTML attachment was scrubbed... URL: From poelzi at poelzi.org Tue Jan 11 06:49:14 2011 From: poelzi at poelzi.org (Daniel Poelzleithner) Date: Tue, 11 Jan 2011 06:49:14 +0100 Subject: [pypy-dev] good by swap of death Message-ID: <4D2BEF5A.4000107@poelzi.org> hi, as I'm surly not the only one having sometimes memory problems when translating pypy. Slowly getting bitten by the swap of death, not registering anything until the desktop stops and rescuing is a very slow process (yes, i have a special root login with /bin/sh shell and memlockd...). I was just able to fix the problem with ulatencyd [1] and as I think translating pypy is a real usecase I would be glad having some feedback. I was quite surprised that after a simple memleak emulator got isolated successfully, pypy translation works like a charm, even when i put additional pressure on the system :-) [1] https://github.com/poelzi/ulatencyd kind regards poelzi -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From fijall at gmail.com Tue Jan 11 07:45:07 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 11 Jan 2011 08:45:07 +0200 Subject: [pypy-dev] good by swap of death In-Reply-To: <4D2BEF5A.4000107@poelzi.org> References: <4D2BEF5A.4000107@poelzi.org> Message-ID: Hey. In general it's next to impossible to translate pypy if swapping. You need at least 2G of RAM for 32bit and 4G for 64bit build. On Tue, Jan 11, 2011 at 7:49 AM, Daniel Poelzleithner wrote: > hi, > > as I'm surly not the only one having sometimes memory problems when > translating pypy. Slowly getting bitten by the swap of death, not > registering anything until the desktop stops and rescuing is a very slow > process (yes, i have a special root login with /bin/sh shell and > memlockd...). > > I was just able to fix the problem with ulatencyd [1] and as I think > translating pypy is a real usecase I would be glad having some feedback. > I was quite surprised that after a simple memleak emulator got isolated > successfully, pypy translation works like a charm, even when i put > additional pressure on the system :-) > > [1] https://github.com/poelzi/ulatencyd > > kind regards > ?poelzi > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From william.leslie.ttg at gmail.com Tue Jan 11 08:43:50 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 11 Jan 2011 18:43:50 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: On 11 January 2011 15:25, Nathanael D. Jones wrote: > Regardng #1: Sandboxing is a major concern for me. Different code will need > different sandboxing levels depending upon who created/approved the code. > I can't have everything in one sandbox - I need isolated boxes on a > per-request level. You will probably want to implement a capability architecture, where the authority to perform some action is bundled together with the method that invokes it. The language E ( http://erights.org ) was designed for this sort of thing (I think it was written for a MUD originally), and would be worth looking at even if you didn't use the language itself. Capability theory is sadly not well known, considering just how well it solves ever more common problems. > The idea is that if functions return the next function to call instead of > calling them, we have explicit tail-call elimination, and we have an > explicit point at which we can rebuild the sandbox and upgrade the code. Right, this is the "driver loop" implementation of TCE. There are several implementations of TCE kicking around, (including one I did for cpython some time ago, a decorator that would patch load ... call* ... return bytecodes with a call to a class that collects the function and arguments, and wraps the function in a driver loop that is bypassed when the target is marked as recursive, too). Unfortunately, using only implicit continuations, once you enter somebody's function, you have no way to get out of it - that person can except: and trap you there if they like. E has a way to protect against this, but you can't protect against the function busy-looping without something more (a way to kill threads, specifically, which probably means taking down the process). > Much data will be user scoped, and therefore lockable. What does lockable mean in this case? Under what conditions does reloading happen while a user continuation is in play? How do you want continuations to interact with the locking? > On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie > wrote: >> >> On 11 January 2011 07:18, Paolo Giarrusso wrote: >> >> The one issue that interests me is where you implement the persistence >> >> boundary - do you go with orthogonal persistence and act as if >> >> everything is preserved, or assume all user code is run within some >> >> sort of (fast and loose) transaction that can be re-entered at will, >> >> providing an API for persistent data access? ?The second case makes >> >> the reloading question a bit more reasonable, because you can always >> >> throw away the current delta and replay the external effects, assuming >> >> the interface for the external events hasn't changed significantly. >> > >> > The key question is: when would you start and commit such transactions? >> > >> > Apart from that, your idea looks very similar to Software >> > Transactional Memory (STM). STM restarts explicitly-marked >> > transactions in a thread when some other thread modifies the affected >> > data (which would be a data race) and commits its transaction. In your >> > case, a transaction is restarted when some other thread modifies the >> > involved code. I don't think it would be useful at all to have regular STM transactions delimited at the room boundary - there are surely going to be other people interacting with the room at the same time as you, and sending everyone else back to the start of the room every time someone modifies the room is going to get old very quickly. The boundary introduced by modifying code is probably significantly more coarse. If you are storing (as the transaction log) only the external effects (user input, random.* output), however, you can build effect commutators automatically. For example, given a code change to object X, you can initialise it with the previous persistent state, and replay the effects within the current transaction. That said, I think having a transaction per room is too coarse - per input or set of inputs makes even more sense. The idea there is that there is much less code to replay when two effects interfere. -- William Leslie From fijall at gmail.com Tue Jan 11 08:48:41 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 11 Jan 2011 09:48:41 +0200 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: Hello. This discussion have drifted considerably. In general this is the pypy development mailing list, so personally I would like to keep discussions only to things that are either very interesting for people actively working on PyPy or have remote possibility that someone will start working on them. Discussing STM without someone being interested in actually looking is out of topic. PS. Feel free to correct me if anyone working actively on pypy is finding this discussion useful for PyPy development. Cheers, fijal On Tue, Jan 11, 2011 at 9:43 AM, William ML Leslie wrote: > On 11 January 2011 15:25, Nathanael D. Jones wrote: >> Regardng #1: Sandboxing is a major concern for me. Different code will need >> different sandboxing levels depending upon who created/approved the code. >> I can't have everything in one sandbox - I need isolated boxes on a >> per-request level. > > You will probably want to implement a capability architecture, where > the authority to perform some action is bundled together with the > method that invokes it. ?The language E ( http://erights.org ) was > designed for this sort of thing (I think it was written for a MUD > originally), and would be worth looking at even if you didn't use the > language itself. ?Capability theory is sadly not well known, > considering just how well it solves ever more common problems. > >> The idea is that if functions return the next function to call instead of >> calling them, we have explicit tail-call elimination, and we have an >> explicit point at which we can rebuild the sandbox and upgrade the code. > > Right, this is the "driver loop" implementation of TCE. ?There are > several implementations of TCE kicking around, (including one I did > for cpython some time ago, a decorator that would patch load ... call* > ... return bytecodes with a call to a class that collects the function > and arguments, and wraps the function in a driver loop that is > bypassed when the target is marked as recursive, too). > > Unfortunately, using only implicit continuations, once you enter > somebody's function, you have no way to get out of it - that person > can except: and trap you there if they like. ?E has a way to protect > against this, but you can't protect against the function busy-looping > without something more (a way to kill threads, specifically, which > probably means taking down the process). > >> Much data will be user scoped, and therefore lockable. > > What does lockable mean in this case? > > Under what conditions does reloading happen while a user continuation > is in play? ?How do you want continuations to interact with the > locking? > >> On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie >> wrote: >>> >>> On 11 January 2011 07:18, Paolo Giarrusso wrote: >>> >> The one issue that interests me is where you implement the persistence >>> >> boundary - do you go with orthogonal persistence and act as if >>> >> everything is preserved, or assume all user code is run within some >>> >> sort of (fast and loose) transaction that can be re-entered at will, >>> >> providing an API for persistent data access? ?The second case makes >>> >> the reloading question a bit more reasonable, because you can always >>> >> throw away the current delta and replay the external effects, assuming >>> >> the interface for the external events hasn't changed significantly. >>> > >>> > The key question is: when would you start and commit such transactions? >>> > >>> > Apart from that, your idea looks very similar to Software >>> > Transactional Memory (STM). STM restarts explicitly-marked >>> > transactions in a thread when some other thread modifies the affected >>> > data (which would be a data race) and commits its transaction. In your >>> > case, a transaction is restarted when some other thread modifies the >>> > involved code. > > I don't think it would be useful at all to have regular STM > transactions delimited at the room boundary - there are surely going > to be other people interacting with the room at the same time as you, > and sending everyone else back to the start of the room every time > someone modifies the room is going to get old very quickly. ?The > boundary introduced by modifying code is probably significantly more > coarse. > > If you are storing (as the transaction log) only the external effects > (user input, random.* output), however, you can build effect > commutators automatically. ?For example, given a code change to object > X, you can initialise it with the previous persistent state, and > replay the effects within the current transaction. > > That said, I think having a transaction per room is too coarse - per > input or set of inputs makes even more sense. ?The idea there is that > there is much less code to replay when two effects interfere. > > -- > William Leslie > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From renesd at gmail.com Tue Jan 11 11:41:58 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 11 Jan 2011 10:41:58 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, did you find this useful? Or would you like me to complete the rest of the module code around it? cheers, On Thu, Jan 6, 2011 at 2:35 PM, Ren? Dudfield wrote: > Hi, > > here is a basic function... without the whole module stuff around it. > I'd need to do the rest of the module another time. > > PyObject* do_long_run (PyObject* self) { > ? ?int i, a, b, c; > ? ?a = 2; > > ? ?Py_BEGIN_ALLOW_THREADS; > ? ?for(i = 0; i < 2000000000; i++) { > ? ? ? ?b = i + a; > ? ? ? ?c = a + i; > ? ?} > ? ?a = c - b; > ? ?Py_END_ALLOW_THREADS; > > ? ?Py_RETURN_NONE; > } > > > // ... > ? ?{ "do_long_run", (PyCFunction) do_long_run , METH_NOARGS, "runs > some cpu intensive code whilst releasing the GIL"}, > // ... > > > > You could test it by calling that function in separate threads. ?With > another thread printing something from python... like in the python > code below. > > > > from threading import Thread > # this assumes the _long_runner is the C module with the do_long_run > function in it. > import _long_runner > > class PythonPrinter(Thread): > ? ?def run(self): > ? ? ? ?for x in range(1000000): > ? ? ? ? ? ?print (x) > > class CLongRunner(Thread): > ? ?def run(self): > ? ? ? ?_long_runner.do_long_run() > > > threads = [PythonPrinter(), CLongRunner()] > for t in threads: > ? ?t.start() > for t in threads: > ? ?t.join() > > > That is untested, and typed without compiling/running... but it should > work, and should give you some insight on how the threading works. > > Maybe you can throw the c function into an existing module for easier > testing... otherwise maybe I can write it in a few days. > > cya. > > > > On Thu, Jan 6, 2011 at 2:13 PM, Amaury Forgeot d'Arc wrote: >> Hi, >> >> 2011/1/6 Ren? Dudfield : >>> Hi, >>> >>> are the thread GIL releasing functions going to be supported in pypy >>> with CPyExt? >>> >>> To allow this usage: >>> >>> PyThreadState *_save; >>> _save = PyEval_SaveThread(); >>> // ...Do some blocking I/O operation, or CPU intensive operation that >>> does not use the python api... >>> PyEval_RestoreThread(_save); >>> >>> Or using the macros... >>> >>> Py_BEGIN_ALLOW_THREADS; >>> // ...Do some blocking I/O operation, or CPU intensive operation that >>> does not use the python api... >>> Py_END_ALLOW_THREADS; >>> >>> Is it not possible or too hard, or is it just not implemented yet? >> >> Frankly, I even don't know. >> At least, the Py_BEGIN_ALLOW_THREADS macros are there and will compile. >> >> OTOH it seems that every call from the pypy interpreter to C releases the GIL, >> and that conversely, every call from to a Python API function will >> grab the GIL for the duration of the call (yes, even Py_INCREF) >> cpyext is maybe already thread-safe, only awfully slow... >> But this is just a guess, from looking at the source code. >> >> I'd be grateful if you could write a custom module and do some tests >> in this area, >> with and without the Py_BEGIN_ALLOW_THREADS calls. >> >> Cheers, >> >> -- >> Amaury Forgeot d'Arc >> > From lac at openend.se Tue Jan 11 12:03:45 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 11 Jan 2011 12:03:45 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: Message from William ML Leslie of "Tue, 11 Jan 2011 18:43:50 +1100." References: Message-ID: <201101111103.p0BB3j9q006294@theraft.openend.se> When I was last in the USA, I met Alessandro Warth, who now works for the Viewpoints Research Institute and whose Phd thesis which you can get on this page: http://www.tinlizzie.org/~awarth/ is about experiments in computer languages to do the sorts of things that you are interested in. He had a neat implementation of something he called 'worlds' to show off, and gave a talk about it. It's chapter 4 of his thesis. It was neat, but I got the distinct impression that the price he was paying for having unlimited cheap undo was that it was going to be quite difficult when users in different worlds, all of whom 'sprouted' from a common, but very distant ancestor and who didn't sprout from each other wanted to share information. Many other people in the room had pressing needs for such things now, and some of them were interested in PyPy. They concluded that what they would really like is a programming language which multiple heaps -- in the sense of chunks of memory that you get when you malloc -- guaranteed to be separate from each other. I couldn't tell them how hard this would be to implement in PyPy. Aside from needing to write a custom garbage collector, what else would be needed? Laura From william.leslie.ttg at gmail.com Tue Jan 11 12:51:55 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 11 Jan 2011 22:51:55 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: <201101111103.p0BB3j9q006294@theraft.openend.se> References: <201101111103.p0BB3j9q006294@theraft.openend.se> Message-ID: Fijal is right, fwiw. There are lots of wonderful things that *could* be implemented, but what we *have* is the stackless transformation and a low-level sandbox system. Discussions of python architecture are always enjoyable, but they don't have a proper place, as far as I know; I often abuse #python for that. On 11 January 2011 22:03, Laura Creighton wrote: > When I was last in the USA, I met Alessandro Warth, who now works for > the Viewpoints Research Institute and whose Phd thesis which you can get > on this page: http://www.tinlizzie.org/~awarth/ is about experiments in > computer languages to do the sorts of things that you are interested in. > He had a neat implementation of something he called 'worlds' to show off, > and gave a talk about it. ?It's chapter 4 of his thesis. I first read this paper as background for pymeta - it's one of those that you just can't put down. There is one thing I felt was missing from the worlds thesis though, namely, handling arrays. To treat them like any other ecmascript object results in strange semantics - if two worlds push to the one array and both are merged, it is as if only one of the pushes occurred. Python lists have different usage patterns than javascript objects, so handling their diffs correctly and efficiently might require some experiments. > They concluded that what they would really like is a programming language > which multiple heaps -- in the sense of chunks of memory that you get > when you malloc -- guaranteed to be separate from each other. ?I couldn't > tell them how hard this would be to implement in PyPy. ?Aside from needing > to write a custom garbage collector, what else would be needed? How would that be different to separate pypy processes? Or is that more like regions in cyclone, or like subinterpreters in TCL? -- William Leslie PS: worlds are an ideal use case for first-class algebraic effects! From amauryfa at gmail.com Tue Jan 11 13:06:57 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 13:06:57 +0100 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, 2011/1/11 Ren? Dudfield : > On Thu, Jan 6, 2011 at 2:35 PM, Ren? Dudfield wrote: >> Hi, >> >> here is a basic function... without the whole module stuff around it. >> I'd need to do the rest of the module another time. >> ... I know how to write extension modules, but I don't have the time nor the interest to do it currently. Would you like to do these experiments yourself, and report us the real status of threads in pypy extensions? -- Amaury Forgeot d'Arc From renesd at gmail.com Tue Jan 11 13:21:16 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 11 Jan 2011 12:21:16 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: cool, ok. On Tue, Jan 11, 2011 at 12:06 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/1/11 Ren? Dudfield : >> On Thu, Jan 6, 2011 at 2:35 PM, Ren? Dudfield wrote: >>> Hi, >>> >>> here is a basic function... without the whole module stuff around it. >>> I'd need to do the rest of the module another time. >>> ... > > I know how to write extension modules, but I don't have the time nor > the interest > to do it currently. > Would you like to do these experiments yourself, and report us the real > status of threads in pypy extensions? > > -- > Amaury Forgeot d'Arc > From holger at merlinux.eu Tue Jan 11 13:58:19 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 11 Jan 2011 13:58:19 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: <20110111125819.GF2145@trillke.net> Hi Maciej, it's not clear to me what posts your are refering to. Certainly not the original post from Nathan? If you think it's a general issue then do a top-level posting and please suggest how "non-active" (defined how?) pypy developers are supposed to know if some topic is interesting without posting about it. And include some concrete rules about when side discussions are ok and when not. Probably all harder to do than improving your ignoring techniques? cheers, holger On Tue, Jan 11, 2011 at 09:48 +0200, Maciej Fijalkowski wrote: > Hello. > > This discussion have drifted considerably. In general this is the pypy > development mailing list, so personally I would like to keep > discussions only to things that are either very interesting for people > actively working on PyPy or have remote possibility that someone will > start working on them. Discussing STM without someone being interested > in actually looking is out of topic. > > PS. Feel free to correct me if anyone working actively on pypy is > finding this discussion useful for PyPy development. > > Cheers, > fijal > > On Tue, Jan 11, 2011 at 9:43 AM, William ML Leslie > wrote: > > On 11 January 2011 15:25, Nathanael D. Jones wrote: > >> Regardng #1: Sandboxing is a major concern for me. Different code will need > >> different sandboxing levels depending upon who created/approved the code. > >> I can't have everything in one sandbox - I need isolated boxes on a > >> per-request level. > > > > You will probably want to implement a capability architecture, where > > the authority to perform some action is bundled together with the > > method that invokes it. ?The language E ( http://erights.org ) was > > designed for this sort of thing (I think it was written for a MUD > > originally), and would be worth looking at even if you didn't use the > > language itself. ?Capability theory is sadly not well known, > > considering just how well it solves ever more common problems. > > > >> The idea is that if functions return the next function to call instead of > >> calling them, we have explicit tail-call elimination, and we have an > >> explicit point at which we can rebuild the sandbox and upgrade the code. > > > > Right, this is the "driver loop" implementation of TCE. ?There are > > several implementations of TCE kicking around, (including one I did > > for cpython some time ago, a decorator that would patch load ... call* > > ... return bytecodes with a call to a class that collects the function > > and arguments, and wraps the function in a driver loop that is > > bypassed when the target is marked as recursive, too). > > > > Unfortunately, using only implicit continuations, once you enter > > somebody's function, you have no way to get out of it - that person > > can except: and trap you there if they like. ?E has a way to protect > > against this, but you can't protect against the function busy-looping > > without something more (a way to kill threads, specifically, which > > probably means taking down the process). > > > >> Much data will be user scoped, and therefore lockable. > > > > What does lockable mean in this case? > > > > Under what conditions does reloading happen while a user continuation > > is in play? ?How do you want continuations to interact with the > > locking? > > > >> On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie > >> wrote: > >>> > >>> On 11 January 2011 07:18, Paolo Giarrusso wrote: > >>> >> The one issue that interests me is where you implement the persistence > >>> >> boundary - do you go with orthogonal persistence and act as if > >>> >> everything is preserved, or assume all user code is run within some > >>> >> sort of (fast and loose) transaction that can be re-entered at will, > >>> >> providing an API for persistent data access? ?The second case makes > >>> >> the reloading question a bit more reasonable, because you can always > >>> >> throw away the current delta and replay the external effects, assuming > >>> >> the interface for the external events hasn't changed significantly. > >>> > > >>> > The key question is: when would you start and commit such transactions? > >>> > > >>> > Apart from that, your idea looks very similar to Software > >>> > Transactional Memory (STM). STM restarts explicitly-marked > >>> > transactions in a thread when some other thread modifies the affected > >>> > data (which would be a data race) and commits its transaction. In your > >>> > case, a transaction is restarted when some other thread modifies the > >>> > involved code. > > > > I don't think it would be useful at all to have regular STM > > transactions delimited at the room boundary - there are surely going > > to be other people interacting with the room at the same time as you, > > and sending everyone else back to the start of the room every time > > someone modifies the room is going to get old very quickly. ?The > > boundary introduced by modifying code is probably significantly more > > coarse. > > > > If you are storing (as the transaction log) only the external effects > > (user input, random.* output), however, you can build effect > > commutators automatically. ?For example, given a code change to object > > X, you can initialise it with the previous persistent state, and > > replay the effects within the current transaction. > > > > That said, I think having a transaction per room is too coarse - per > > input or set of inputs makes even more sense. ?The idea there is that > > there is much less code to replay when two effects interfere. > > > > -- > > William Leslie > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- From fijall at gmail.com Tue Jan 11 14:10:58 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 11 Jan 2011 15:10:58 +0200 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: <20110111125819.GF2145@trillke.net> References: <20110111125819.GF2145@trillke.net> Message-ID: On Tue, Jan 11, 2011 at 2:58 PM, holger krekel wrote: > Hi Maciej, > > it's not clear to me what posts your are refering to. Certainly > not the original post from Nathan? > > If you think it's a general issue then do a top-level posting and please > suggest how "non-active" (defined how?) pypy developers are supposed to > know if some topic is interesting without posting about it. ?And include > some concrete rules about when side discussions are ok and when not. > Probably all harder to do than improving your ignoring techniques? Well, I can easily ignore things that are not interesting to me. My point was that discussions about topics (like capabilities) only make sense if there is someone even remotely interested in implementing that in PyPy. Otherwise discussions tend to drift into "but that's also a cool idea", which is off-topic for PyPy, since noone actually wants to implement any of those. For example I would say all GIL-removal discussions are offtopic unless someone has a will to experiment with implementing it. That's, as I said above, my personal opinion, backed by a fact that this is the list for discussing developing PyPy (so some people, like me, feel obliged to read whole discussions here). Feel free to propose some other guidelines for what is on-topic and what is off-topic for that list. As active I mean anyone doing any work on PyPy. Be it a typo in documentation :) Cheers, fijal > > cheers, > holger > > On Tue, Jan 11, 2011 at 09:48 +0200, Maciej Fijalkowski wrote: >> Hello. >> >> This discussion have drifted considerably. In general this is the pypy >> development mailing list, so personally I would like to keep >> discussions only to things that are either very interesting for people >> actively working on PyPy or have remote possibility that someone will >> start working on them. Discussing STM without someone being interested >> in actually looking is out of topic. >> >> PS. Feel free to correct me if anyone working actively on pypy is >> finding this discussion useful for PyPy development. >> >> Cheers, >> fijal >> >> On Tue, Jan 11, 2011 at 9:43 AM, William ML Leslie >> wrote: >> > On 11 January 2011 15:25, Nathanael D. Jones wrote: >> >> Regardng #1: Sandboxing is a major concern for me. Different code will need >> >> different sandboxing levels depending upon who created/approved the code. >> >> I can't have everything in one sandbox - I need isolated boxes on a >> >> per-request level. >> > >> > You will probably want to implement a capability architecture, where >> > the authority to perform some action is bundled together with the >> > method that invokes it. ?The language E ( http://erights.org ) was >> > designed for this sort of thing (I think it was written for a MUD >> > originally), and would be worth looking at even if you didn't use the >> > language itself. ?Capability theory is sadly not well known, >> > considering just how well it solves ever more common problems. >> > >> >> The idea is that if functions return the next function to call instead of >> >> calling them, we have explicit tail-call elimination, and we have an >> >> explicit point at which we can rebuild the sandbox and upgrade the code. >> > >> > Right, this is the "driver loop" implementation of TCE. ?There are >> > several implementations of TCE kicking around, (including one I did >> > for cpython some time ago, a decorator that would patch load ... call* >> > ... return bytecodes with a call to a class that collects the function >> > and arguments, and wraps the function in a driver loop that is >> > bypassed when the target is marked as recursive, too). >> > >> > Unfortunately, using only implicit continuations, once you enter >> > somebody's function, you have no way to get out of it - that person >> > can except: and trap you there if they like. ?E has a way to protect >> > against this, but you can't protect against the function busy-looping >> > without something more (a way to kill threads, specifically, which >> > probably means taking down the process). >> > >> >> Much data will be user scoped, and therefore lockable. >> > >> > What does lockable mean in this case? >> > >> > Under what conditions does reloading happen while a user continuation >> > is in play? ?How do you want continuations to interact with the >> > locking? >> > >> >> On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie >> >> wrote: >> >>> >> >>> On 11 January 2011 07:18, Paolo Giarrusso wrote: >> >>> >> The one issue that interests me is where you implement the persistence >> >>> >> boundary - do you go with orthogonal persistence and act as if >> >>> >> everything is preserved, or assume all user code is run within some >> >>> >> sort of (fast and loose) transaction that can be re-entered at will, >> >>> >> providing an API for persistent data access? ?The second case makes >> >>> >> the reloading question a bit more reasonable, because you can always >> >>> >> throw away the current delta and replay the external effects, assuming >> >>> >> the interface for the external events hasn't changed significantly. >> >>> > >> >>> > The key question is: when would you start and commit such transactions? >> >>> > >> >>> > Apart from that, your idea looks very similar to Software >> >>> > Transactional Memory (STM). STM restarts explicitly-marked >> >>> > transactions in a thread when some other thread modifies the affected >> >>> > data (which would be a data race) and commits its transaction. In your >> >>> > case, a transaction is restarted when some other thread modifies the >> >>> > involved code. >> > >> > I don't think it would be useful at all to have regular STM >> > transactions delimited at the room boundary - there are surely going >> > to be other people interacting with the room at the same time as you, >> > and sending everyone else back to the start of the room every time >> > someone modifies the room is going to get old very quickly. ?The >> > boundary introduced by modifying code is probably significantly more >> > coarse. >> > >> > If you are storing (as the transaction log) only the external effects >> > (user input, random.* output), however, you can build effect >> > commutators automatically. ?For example, given a code change to object >> > X, you can initialise it with the previous persistent state, and >> > replay the effects within the current transaction. >> > >> > That said, I think having a transaction per room is too coarse - per >> > input or set of inputs makes even more sense. ?The idea there is that >> > there is much less code to replay when two effects interfere. >> > >> > -- >> > William Leslie >> > _______________________________________________ >> > pypy-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/pypy-dev >> > >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > -- > From Arnd.Rechenburg at tomtom.com Tue Jan 11 15:02:03 2011 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Tue, 11 Jan 2011 15:02:03 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" Message-ID: Hi, Could someone help me to avoid the following exception? Exception "OSError: [Errno 10] No child processes" in method __del__ of ', mode 'r' at 0x00002b58059f05f8> ignored Thanks in advance, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Tue Jan 11 15:33:46 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 15:33:46 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi, 2011/1/11 Arnd Rechenburg : > Could someone help me to avoid the following exception? > > Exception "OSError: [Errno 10] No child processes" in method __del__ of > ', mode 'r' at 0x00002b58059f05f8> ignored More context would be useful: which version of pypy are you using, what are you trying to do, are there some kind of subprocesses, etc etc. -- Amaury Forgeot d'Arc From Arnd.Rechenburg at tomtom.com Tue Jan 11 16:04:10 2011 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Tue, 11 Jan 2011 16:04:10 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi Amaury, I use the following environment: Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48) [PyPy 1.4.1] on linux2 For the moment I use os.execv and os.waitpid(-1, os.WNOHANG) to start and handle parallel jobs by myself. I tried already a 'try except OSError' around os.waitpid but without success. Greetings, Arnd -----Original Message----- From: Amaury Forgeot d'Arc [mailto:amauryfa at gmail.com] Sent: Dienstag, 11. Januar 2011 15:34 To: Arnd Rechenburg Cc: pypy-dev at codespeak.net Subject: Re: [pypy-dev] "OSError: [Errno 10] No child processes" Hi, 2011/1/11 Arnd Rechenburg : > Could someone help me to avoid the following exception? > > Exception "OSError: [Errno 10] No child processes" in method __del__ of > ', mode 'r' at 0x00002b58059f05f8> ignored More context would be useful: which version of pypy are you using, what are you trying to do, are there some kind of subprocesses, etc etc. -- Amaury Forgeot d'Arc From santagada at gmail.com Tue Jan 11 16:25:27 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 11 Jan 2011 13:25:27 -0200 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: <20110111125819.GF2145@trillke.net> Message-ID: On Tue, Jan 11, 2011 at 11:10 AM, Maciej Fijalkowski wrote: > On Tue, Jan 11, 2011 at 2:58 PM, holger krekel wrote: >> Hi Maciej, >> >> it's not clear to me what posts your are refering to. Certainly >> not the original post from Nathan? >> >> If you think it's a general issue then do a top-level posting and please >> suggest how "non-active" (defined how?) pypy developers are supposed to >> know if some topic is interesting without posting about it. ?And include >> some concrete rules about when side discussions are ok and when not. >> Probably all harder to do than improving your ignoring techniques? > > Well, I can easily ignore things that are not interesting to me. > > My point was that discussions about topics (like capabilities) only > make sense if there is someone even remotely interested in > implementing that in PyPy. Otherwise discussions tend to drift into > "but that's also a cool idea", which is off-topic for PyPy, since > noone actually wants to implement any of those. Well we could tell him how to implement capabilities in pypy and how easy/hard this seems. Without looking much at it I think you could implement it by creating a new object space, like those: http://codespeak.net/pypy/dist/pypy/doc/objspace-proxies.html > For example I would say all GIL-removal discussions are offtopic > unless someone has a will to experiment with implementing it. This is also something I see as a problem in communication, there should be at least a FAQ question about this or even a good description of how pypy does Threads and why it does the same thing as CPython. > That's, as I said above, my personal opinion, backed by a fact that > this is the list for discussing developing PyPy (so some people, like > me, feel obliged to read whole discussions here). Feel free to propose > some other guidelines for what is on-topic and what is off-topic for > that list. Maybe pypy developers could steers discussions into being on topic. > As active I mean anyone doing any work on PyPy. Be it a typo in documentation :) People feel that they are helping by discussing on the list ideas and approaches that maybe the pypy team might not know. > Cheers, > fijal > -- Leonardo Santagada From amauryfa at gmail.com Tue Jan 11 16:41:47 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 16:41:47 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: 2011/1/11 Arnd Rechenburg : > Hi Amaury, > > I use the following environment: > > Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48) > [PyPy 1.4.1] on linux2 > > For the moment I use os.execv and os.waitpid(-1, os.WNOHANG) to start > and handle parallel jobs by myself. > > I tried already a 'try except OSError' around os.waitpid but without > success. It's probably related to our file.__del__ which does not silence errors raised by close(). Could you file a ticket about this on the issue tracker? with a small test case if possible. https://codespeak.net/issue/pypy-dev/ -- Amaury Forgeot d'Arc From nathanael.jones at gmail.com Tue Jan 11 16:48:53 2011 From: nathanael.jones at gmail.com (Nathanael D. Jones) Date: Tue, 11 Jan 2011 10:48:53 -0500 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: <201101111103.p0BB3j9q006294@theraft.openend.se> Message-ID: @fijal: I agree that parts of this topic have drifted off-topic for the py-py dev mailing list, so I've created a new mailing list specifically for discussions of collaboratively-edited software. I'm inviting everyone to join the new mailing list at collaborative-games at googlegroups.com As far as the discussion regarding PyPy's capabilities with multiple-sandboxing, reloading, and continuation serialization go, I think that is a relevant discussion for this forum. I'm still very interested in understanding (a) what PyPy can already do, and (b) what is involved in adding the features I need. I'm itching to work on PyPy :) @Laura + William: Alessandro's paper concisely describes what I was planning on doing, with the exception that the sprout() method was taking place in Java. After reading the paper I see the utility of also allowing sprouting within Javascript. I am a bit concerned about data size, however. Loading all world data is an expensive op, even if it creates a simple way to perform psuedo-transactions. Nathanael http://nathanaeljones.com On Tue, Jan 11, 2011 at 6:51 AM, William ML Leslie < william.leslie.ttg at gmail.com> wrote: > Fijal is right, fwiw. There are lots of wonderful things that *could* > be implemented, but what we *have* is the stackless transformation and > a low-level sandbox system. Discussions of python architecture are > always enjoyable, but they don't have a proper place, as far as I > know; I often abuse #python for that. > > On 11 January 2011 22:03, Laura Creighton wrote: > > When I was last in the USA, I met Alessandro Warth, who now works for > > the Viewpoints Research Institute and whose Phd thesis which you can get > > on this page: http://www.tinlizzie.org/~awarth/ is about experiments in > > computer languages to do the sorts of things that you are interested in. > > He had a neat implementation of something he called 'worlds' to show off, > > and gave a talk about it. It's chapter 4 of his thesis. > > I first read this paper as background for pymeta - it's one of those > that you just can't put down. There is one thing I felt was missing > from the worlds thesis though, namely, handling arrays. To treat them > like any other ecmascript object results in strange semantics - if two > worlds push to the one array and both are merged, it is as if only one > of the pushes occurred. Python lists have different usage patterns > than javascript objects, so handling their diffs correctly and > efficiently might require some experiments. > > > They concluded that what they would really like is a programming language > > which multiple heaps -- in the sense of chunks of memory that you get > > when you malloc -- guaranteed to be separate from each other. I couldn't > > tell them how hard this would be to implement in PyPy. Aside from > needing > > to write a custom garbage collector, what else would be needed? > > How would that be different to separate pypy processes? Or is that > more like regions in cyclone, or like subinterpreters in TCL? > > -- > William Leslie > > PS: worlds are an ideal use case for first-class algebraic effects! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfbolz at gmx.de Tue Jan 11 17:23:17 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 11 Jan 2011 17:23:17 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: <201101111103.p0BB3j9q006294@theraft.openend.se> Message-ID: <4D2C83F5.8010202@gmx.de> On 01/11/2011 04:48 PM, Nathanael D. Jones wrote: > As far as the discussion regarding PyPy's capabilities with > multiple-sandboxing, reloading, and continuation serialization go, I > think that is a relevant discussion for this forum. > > I'm still very interested in understanding (a) what PyPy can already do, > and (b) what is involved in adding the features I need. I'm itching to > work on PyPy :) PyPy implements two things that are related somehow: 1) process sandboxing, which is just a way to control what one instance of an interpreter is allowed to do in terms of system calls, CPU and memory used. This cannot be controlled in a fine-grained way, every sandbox needs to be its own process, and communication is not implemented yet. 2) "stackless", which is a way for interpreters to get some control over the C stack, including being able to artificially create one. This does not mean that you get serialization for free, it's just that we implemented it carefully for Python. To go further, I really think you need careful language design to get the capability model and the versioning/updating right. IMO you won't be able to extend standard Python, the semantics of it are too complex to shoehorn something with (so far) vague goals on top of it. This language design is partially orthogonal to what PyPy provides you with. After thinking about what your language should look like, you can consider to use PyPy as an implementation platform for it, or not. Cheers, Carl Friedrich From arigo at tunes.org Tue Jan 11 21:56:53 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 11 Jan 2011 21:56:53 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi Amaury, On Tue, Jan 11, 2011 at 4:41 PM, Amaury Forgeot d'Arc wrote: > It's probably related to our file.__del__ which does not silence > errors raised by close(). Indeed, it seems to be the case. Arnd, to answer the original question: there is a file that you are not explicitly closing, and when its __del__ method is (later) invoked, it raises an OSError which is printed and ignored. The real problem you have there is that you should close your files (in this case, an file, so I guess it's actually a pipe returned by os.popen*()). The secondary problem that we have is what to do when file.__del__ gets an exception from calling close(). Is it ok to just ignore it? FWIW in CPython, file.__del__ also prints an error message if close() raises an exception. So from that point of view there is nothing to fix in PyPy, apart maybe making the error message a bit more explicit (CPython prints "close failed in file object destructor:..."). A bient?t, Armin. From amauryfa at gmail.com Tue Jan 11 22:09:21 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 22:09:21 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: 2011/1/11 Armin Rigo : > The secondary problem that we have is what to do when file.__del__ > gets an exception from calling close(). ?Is it ok to just ignore it? > FWIW in CPython, file.__del__ also prints an error message if close() > raises an exception. ?So from that point of view there is nothing to > fix in PyPy, apart maybe making the error message a bit more explicit > (CPython prints "close failed in file object destructor:..."). The io module chose to silence the error; I suggest to do the same, even if the reasoning is different from the comment below:: class IOBase: def __del__(self): """Destructor. Calls close().""" # The try/except block is in case this is called at program # exit time, when it's possible that globals have already been # deleted, and then the close() call might fail. Since # there's nothing we can do about such failures and they annoy # the end users, we suppress the traceback. try: self.close() except: pass -- Amaury Forgeot d'Arc From p.giarrusso at gmail.com Wed Jan 12 02:49:10 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Wed, 12 Jan 2011 02:49:10 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: On Tue, Jan 11, 2011 at 22:09, Amaury Forgeot d'Arc wrote: > 2011/1/11 Armin Rigo : >> The secondary problem that we have is what to do when file.__del__ >> gets an exception from calling close(). ?Is it ok to just ignore it? >> FWIW in CPython, file.__del__ also prints an error message if close() >> raises an exception. ?So from that point of view there is nothing to >> fix in PyPy, apart maybe making the error message a bit more explicit >> (CPython prints "close failed in file object destructor:..."). > > The io module chose to silence the error; > I suggest to do the same, even if the reasoning is different > from the comment below:: > > class IOBase: > ? ?def __del__(self): > ? ? ? ?"""Destructor. ?Calls close().""" > ? ? ? ?# The try/except block is in case this is called at program > ? ? ? ?# exit time, when it's possible that globals have already been > ? ? ? ?# deleted, and then the close() call might fail. ?Since > ? ? ? ?# there's nothing we can do about such failures and they annoy > ? ? ? ?# the end users, we suppress the traceback. I propose that PyPy keeps reporting the error for files opened in any write mode, because then an error on close might mean that an I/O error happened (it is possible in some scenarios on Linux, see below [1]); the programmer thus need to explicitly close the file and check for exceptions. Having writable files closed by __del__ also sounds like a bad idea if any buffering is involved, because close() time becomes unpredictable. Those are mostly arguments against __del__ _silently_ calling close(); but when the potential bug is triggered, i.e. you hit an I/O error, it's even more important to report it to the user. I also propose that any error is ignored for read-mode files - because then why should we care for it? The original error message talked about a read-mode file handle. [1] From the close(2) man page: "Not checking the return value of close() is a common but nevertheless serious programming error. It is quite possible that errors on a previous write(2) operation are first reported at the final close(). Not checking the return value when closing the file may lead to silent loss of data. This can especially be observed with NFS and with disk quota." Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From arigo at tunes.org Thu Jan 13 13:12:42 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 13 Jan 2011 13:12:42 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi Amaury, On Wed, Jan 12, 2011 at 2:49 AM, Paolo Giarrusso wrote: > I propose that PyPy keeps reporting the error for files opened in any > write mode I would also think that it's better to keep reporting errors (and for all files instead of just write mode files). In my opinion, it would, if nothing else, give users the message: "we got an error when close()ing this file automatically, but you should really close it explicitly yourself in the first place". Maybe it can even be written in that sense in the error message. A bient?t, Armin. From fijall at gmail.com Thu Jan 13 13:25:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 13 Jan 2011 14:25:49 +0200 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: On Thu, Jan 13, 2011 at 2:12 PM, Armin Rigo wrote: > Hi Amaury, > > On Wed, Jan 12, 2011 at 2:49 AM, Paolo Giarrusso wrote: >> I propose that PyPy keeps reporting the error for files opened in any >> write mode > > I would also think that it's better to keep reporting errors (and for > all files instead of just write mode files). ?In my opinion, it would, > if nothing else, give users the message: "we got an error when > close()ing this file automatically, but you should really close it > explicitly yourself in the first place". ?Maybe it can even be written > in that sense in the error message. > How about a link to differencies between pypy and cpython, especially about closing files? Cheers, fijal > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Thu Jan 13 13:43:40 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 13 Jan 2011 13:43:40 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi, On Thu, Jan 13, 2011 at 1:25 PM, Maciej Fijalkowski wrote: >> close()ing this file automatically, but you should really close it >> explicitly yourself in the first place". ?Maybe it can even be written >> in that sense in the error message. > > How about a link to differencies between pypy and cpython, especially > about closing files? Yes please :-) Armin From p.giarrusso at gmail.com Thu Jan 13 13:49:50 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Thu, 13 Jan 2011 13:49:50 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: On Thu, Jan 13, 2011 at 13:43, Armin Rigo wrote: > Hi, > > On Thu, Jan 13, 2011 at 1:25 PM, Maciej Fijalkowski wrote: >>> close()ing this file automatically, but you should really close it >>> explicitly yourself in the first place". ?Maybe it can even be written >>> in that sense in the error message. >> How about a link to differencies between pypy and cpython, especially >> about closing files? And I guess you can also mention Jython and IronPython as being PyPy-like (if the behavior just depends on having a GC instead of refcounting). Disclaimer: I've never used IronPython. > Yes please :-) -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From bhartsho at yahoo.com Sat Jan 15 03:09:24 2011 From: bhartsho at yahoo.com (Hart's Antler) Date: Fri, 14 Jan 2011 18:09:24 -0800 (PST) Subject: [pypy-dev] PyPy on Android Message-ID: <493797.19850.qm@web114002.mail.gq1.yahoo.com> Hi all PyPy-devs, I'm not sure if my error is simply a 64bit issue, and it would work from 32bit linux. ?Can i simply -DMAX_LONG not to be 64bit and then using the Android NDK gcc (which i think is 32bits) is ok? ?Or should i recompile Android's NDK gcc so that its 64bits? Almost working example:http://pastebin.com/bjwEh8E7 Error:/home/brett/android-ndk-r5/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gcc -MMD -MP -MF /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__ ?-Wno-psabi -march=armv5te -mtune=xscale -msoft-float -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -I/tmp/usession-default-6/jni -DANDROID -I/home/brett/RPythonic/pypy/pypy/translator/c -DPYPY_STANDALONE -Wa,--noexecstack -O2 -DNDEBUG -g -I/home/brett/android-ndk-r5/platforms/android-3/arch-arm/usr/include -c ?/tmp/usession-default-6/jni/testing_1.c -o /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.oIn file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_prerequisite.h:7,?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/common_header.h:44,?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:1:/home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:58:6: error: #error "error in LONG_MAX (64-bit sources but a 32-bit compiler?)"/home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:61:6: error: #error "unsupported value for LONG_MIN"/tmp/usession-default-6/jni/testing_1.c:85: error: expected specifier-qualifier-list before 'PyObject'/tmp/usession-default-6/jni/testing_1.c:404: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token/tmp/usession-default-6/jni/testing_1.c:409: error: expected ')' before '*' tokenIn file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_include.h:58,?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:564: -brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.giarrusso at gmail.com Sat Jan 15 12:42:19 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Sat, 15 Jan 2011 12:42:19 +0100 Subject: [pypy-dev] PyPy on Android In-Reply-To: <493797.19850.qm@web114002.mail.gq1.yahoo.com> References: <493797.19850.qm@web114002.mail.gq1.yahoo.com> Message-ID: On Sat, Jan 15, 2011 at 03:09, Hart's Antler wrote: > > Hi all PyPy-devs, > I'm not sure if my error is simply a 64bit issue, and it would work from 32bit linux. ?Can i simply -DMAX_LONG not to be 64bit and then using the Android NDK gcc (which i think is 32bits) is ok? I guess MAX_LONG is calculated internally. You probably need to use a different Python interpreter, a 32bit one, to build PyPy (instructions on the website). If none is available, you need to build one from scratch. Given that you're cross-compiling PyPy, I wonder whether Python on ARM (the one you'd use in a non-cross-platform build) is any different from Python on x86, and if this difference is important; surely it is different from Python on x86_64, as you noticed. > Or should i recompile Android's NDK gcc so that its 64bits? Maybe I missed something, but is that even possible? Your toolchain seems to target ARM, and according to common sense and Wikipedia [1] there are no 64bit ARM processors. It seems that Android supports other target architectures, even x86 it seems, still I doubt the existence of 64bit phones :-D. [1] http://en.wikipedia.org/wiki/ARM_architecture > Almost working example: > http://pastebin.com/bjwEh8E7 > Error: > /home/brett/android-ndk-r5/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gcc -MMD -MP -MF /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__ ?-Wno-psabi -march=armv5te -mtune=xscale -msoft-float -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -I/tmp/usession-default-6/jni -DANDROID -I/home/brett/RPythonic/pypy/pypy/translator/c -DPYPY_STANDALONE -Wa,--noexecstack -O2 -DNDEBUG -g -I/home/brett/android-ndk-r5/platforms/android-3/arch-arm/usr/include -c ?/tmp/usession-default-6/jni/testing_1.c -o /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.o > In file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_prerequisite.h:7, > ?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/common_header.h:44, > ?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:1: > /home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:58:6: error: #error "error in LONG_MAX (64-bit sources but a 32-bit compiler?)" > /home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:61:6: error: #error "unsupported value for LONG_MIN" > /tmp/usession-default-6/jni/testing_1.c:85: error: expected specifier-qualifier-list before 'PyObject' > /tmp/usession-default-6/jni/testing_1.c:404: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token > /tmp/usession-default-6/jni/testing_1.c:409: error: expected ')' before '*' token > In file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_include.h:58, > ?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:564: > > -brett > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From konryd at gmail.com Mon Jan 17 12:30:29 2011 From: konryd at gmail.com (Konrad Delong) Date: Mon, 17 Jan 2011 12:30:29 +0100 Subject: [pypy-dev] In search of references Message-ID: Hi all, I am currently finishing up my master thesis. In it, I'd like to mention PyPy, but I think I need references to a couple of statements in order... well... not to lie :) So here it is: 1. PyPy is based on an approach first used in implementing Smalltalk-80 (the idea of implementing a dynamic language in its own subset, then statically analyzing and compiling the interpreter) 2. The main reason PyPy decided to implement GIL was for the c extensions to run unchanged. I obviously don't want you to do my job for me, but I searched a lot on this topic already, and any hint where to look further would be appreciated. cheers, Konrad From cfbolz at gmx.de Mon Jan 17 13:13:29 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 17 Jan 2011 13:13:29 +0100 Subject: [pypy-dev] In search of references In-Reply-To: References: Message-ID: <4D343269.1060905@gmx.de> On 01/17/2011 12:30 PM, Konrad Delong wrote: > Hi all, > > I am currently finishing up my master thesis. In it, I'd like to > mention PyPy, but I think I need references to a couple of statements > in order... well... not to lie :) > > So here it is: > > 1. PyPy is based on an approach first used in implementing > Smalltalk-80 (the idea of implementing a dynamic language in its own > subset, then statically analyzing and compiling the interpreter) Read this: http://portal.acm.org/citation.cfm?id=1176753 It has a discussion about the relation to Squeak at the end. > 2. The main reason PyPy decided to implement GIL was for the c > extensions to run unchanged. There is no "official" reference for this. The reason why we have a GIL is because doing anything else is significantly harder, and would require a GC that works well with threads. This is a non-trivial task. In general, if you are looking for more "scientific" reference, look here: http://codespeak.net/pypy/dist/pypy/doc/extradoc.html There is also an article about to be published about the main JIT optimization here: http://codespeak.net/svn/pypy/extradoc/talk/pepm2011/bolz-allocation-removal.pdf Cheers, Carl Friedrich From p.giarrusso at gmail.com Mon Jan 17 13:40:22 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 17 Jan 2011 13:40:22 +0100 Subject: [pypy-dev] In search of references In-Reply-To: References: Message-ID: On Mon, Jan 17, 2011 at 12:30, Konrad Delong wrote: > Hi all, > > I am currently finishing up my master thesis. In it, I'd like to > mention PyPy, but I think I need references to a couple of statements > in order... well... not to lie :) > > So here it is: > > 1. PyPy is based on an approach first used in implementing > Smalltalk-80 (the idea of implementing a dynamic language in its own > subset, then statically analyzing and compiling the interpreter) > 2. The main reason PyPy decided to implement GIL was for the c > extensions to run unchanged. Beyond what Carl wrote, no C extension ran unchanged until the introduction of cpyext in 2010: http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html Bye -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From jacob at openend.se Thu Jan 20 16:16:44 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Thu, 20 Jan 2011 16:16:44 +0100 Subject: [pypy-dev] Branches in the mercurial repository Message-ID: <201101201616.46698.jacob@openend.se> I have made a cleanup of branches in the mercurial repository, closing about 40 of them. We currently have 34 open branches and I think that some of them may no longer be useful. Please check the branches you are responsible for and close them if they are no longer needed. By keeping the repository free from cruft we make it a lot easier for newcomers to navigate. The procedure for closing a branch is simple: hg up -C hg commit --close-branch -m 'Closing disused branch.' hg up -C default # switch back to "good" branch Here is a list of currently open branches: Alex mixed-submodules Mon Jan 03 12:50:00 2011 Alexander sepcomp Wed Dec 09 12:15:29 2009 Amaury build-external Tue Jun 10 10:48:46 2008 Amaury cpyext-init-cleanup Wed Jun 09 15:56:49 2010 Amaury separate-compilation Fri Jan 22 17:32:50 2010 Amaury unicode_filename Fri Apr 24 16:17:08 2009 Amaury unicode_filename-2 Fri Jul 16 20:39:39 2010 Antonio cli-jit Wed Jan 20 09:09:23 2010 Antonio jitypes2 Thu Jan 20 10:16:27 2011 Armin 32ptr-on-64bit Fri Oct 15 13:13:44 2010 Armin inline-shadowstack Mon Dec 06 10:46:52 2010 Armin jit-longlong Fri Jan 14 16:17:13 2011 Armin jit-lsprofile Wed Jan 05 18:26:14 2011 Armin jit-tagged Mon Dec 20 10:36:09 2010 Bartosz fast-ctypes Thu Aug 19 11:35:15 2010 Carl bridges-experimental Thu Jan 21 15:33:18 2010 Carl reflex-support Tue Jan 11 14:32:08 2011 Carl unroll-safe-if-const-arg Tue Jan 26 15:36:20 2010 Dan gdbm Thu Nov 25 00:00:30 2010 Daniel micronumpy Sun Sep 12 08:59:28 2010 Daniel micronumpy-resync Tue Oct 19 05:19:25 2010 Daniel ootype-virtualrefs Sun Nov 07 19:39:49 2010 Daniel psycopg2compatibility Thu Jan 13 16:05:44 2011 David arm-backed-float Mon Jan 17 17:31:39 2011 David arm-backend-2 Fri Oct 08 01:35:03 2010 Hakan jit-short-preamble Mon Jan 17 07:33:06 2011 Jean-Philippe avm Mon Jul 26 22:47:11 2010 Maciej guard-improvements Tue Jan 11 15:49:26 2011 Maciej out-of-line-guards Mon Jan 17 19:00:12 2011 Michael bytearray Thu Jan 20 13:44:58 2011 Samuele ctypes-stable Fri Apr 11 15:44:57 2008 Samuele run-django Sat Jul 12 09:46:07 2008 Travis jit-stackless Mon Apr 05 16:48:39 2010 holger pytest2 Wed Jan 19 19:24:33 2011 Jacob Hall?n From samuele.pedroni at gmail.com Thu Jan 20 17:33:39 2011 From: samuele.pedroni at gmail.com (Samuele Pedroni) Date: Thu, 20 Jan 2011 17:33:39 +0100 Subject: [pypy-dev] Branches in the mercurial repository In-Reply-To: <201101201616.46698.jacob@openend.se> References: <201101201616.46698.jacob@openend.se> Message-ID: > Samuele ? ? ? ? ? ? ?ctypes-stable ? ? ? ? ? ?Fri Apr 11 15:44:57 2008 > Samuele ? ? ? ? ? ? ?run-django ? ? ? ? ? ? ? Sat Jul 12 09:46:07 2008 these corresponded to results of the Google contracts, whether the svn repo is staying around read-only or going away, they were referred through urls to the svn repo of course. I suppose they are past their material usefulness, I suppose they could be closed, don't know if people have a different opinion on this. Samuele From alex.gaynor at gmail.com Thu Jan 20 17:42:51 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 20 Jan 2011 10:42:51 -0600 Subject: [pypy-dev] Branches in the mercurial repository In-Reply-To: References: <201101201616.46698.jacob@openend.se> Message-ID: On Thu, Jan 20, 2011 at 10:33 AM, Samuele Pedroni wrote: > > Samuele ctypes-stable Fri Apr 11 15:44:57 2008 > > Samuele run-django Sat Jul 12 09:46:07 2008 > > these corresponded to results of the Google contracts, whether the svn > repo is staying around read-only or going away, they were referred > through urls to the svn repo of course. I suppose they are past their > material usefulness, I suppose they could be closed, don't know if > people have a different opinion on this. > > Samuele > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > closed just means that they are no longer being used (and won't be shown in the default list of branches), I tihnk those are safe to close. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobami at googlemail.com Thu Jan 20 22:00:41 2011 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 20 Jan 2011 22:00:41 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org Message-ID: Hi all, I want to announce the release of Codespeed 0.7, the version which is now powering speed.pypy.org The announcement has been made on http://groups.google.com/group/codespeed/browse_thread/thread/3dd2d9491b26d044, but I will paste it here: -------------------------------------------------- I will briefly describe the most significant improvements. Result Reports -------------- While the previous Codespeed version allowed a project to analyse and visualize performance changes and compare different versions or even "competitors", it didn't address well a use-case: a core developer that is interested in the daily performance variations. You had to go to the Changes view, click on the desired revision, maybe another executable, and again for every revision you wanted to check. That use case is now addressed by Reports. They are a summary of the most significant changes on every run. It's a ultra-condensed Changes table. It is shown on the main page, and you can even subscribe to a RSS feed. That way you only visit the page when you are interested in some particular revision or change. After a few weeks of use, you may find out that a different "algorithm" is better (for example dropping trends or trends triggering the red state...), or that the thresholds need to be tweaked. And there are surely other interesting run statistics that could be gathered and shown. I eagerly await the feedback and ensuing discussion ;-) Support for small screen sizes ------------------------------ Using CSS media queries, the site layout adapts to smaller screens. It is optimized for 3 sizes: - Desktop: normal layout for big screens - Netbook, tablets (~1000px): Similar layout but smaller margins, smaller plots. Coincidentally, it is about the same width of Maciej's laptop screen ;-) - Smartphones: iPhones and Androids in landscape mode should be now able to visit the site without problems. Though it can be tested just by resizing the browser window, it would be great to get some feedback of anyone actually using the above mentioned devices! Performance improvements ------------------------ Exarkun made the timeline view faster by pre-fetching all related data. The timeline grid took 1.46 seconds to load, and a single plot 0.45 seconds. They now need 1.26 and 0.26 seconds respectively. A big improvement! There is yet another improvement waiting in the wings, but we still need to figure out some regression it introduces. The Changes view is also much faster (from 480ms to 210ms), because it now uses the cached table stored in the corresponding report (only when viewing with the default trend=10 ). A last detail --------------- Another change you may notice is that the default checked executables in the Comparison view are now configurable (I didn't forget Maciej! :). So they are not all selected per default any more, so the plot shouldn't look any longer like the Flying Spaghetti Monster ;-) I'll leave it there. I hope you enjoy the changes! Miquel From fijall at gmail.com Thu Jan 20 23:06:15 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 21 Jan 2011 00:06:15 +0200 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: Message-ID: Great! On Thu, Jan 20, 2011 at 11:00 PM, Miquel Torres wrote: > Hi all, > > I want to announce the release of Codespeed 0.7, the version which is > now powering speed.pypy.org > > The announcement has been made on > http://groups.google.com/group/codespeed/browse_thread/thread/3dd2d9491b26d044, > but I will paste it here: > > -------------------------------------------------- > ?I will briefly describe the most significant improvements. > > > Result Reports > -------------- > While the previous Codespeed version allowed a project to analyse and > visualize performance changes and compare different versions or even > "competitors", it didn't address well a use-case: a core developer > that is interested in the daily performance variations. > > You had to go to the Changes view, click on the desired revision, > maybe another executable, and again for every revision you wanted to > check. > > That use case is now addressed by Reports. They are a summary of the > most significant changes on every run. It's a ultra-condensed Changes > table. It is shown on the main page, and you can even subscribe to a > RSS feed. That way you only visit the page when you are interested in > some particular revision or change. > > After a few weeks of use, you may find out that a different > "algorithm" is better (for example dropping trends or trends > triggering the red state...), or that the thresholds need to be > tweaked. And there are surely other interesting run statistics that > could be gathered and shown. I eagerly await the feedback and ensuing > discussion ;-) > > > Support for small screen sizes > ------------------------------ > > Using CSS media queries, the site layout adapts to smaller screens. It > is optimized for 3 sizes: > - Desktop: normal layout for big screens > - Netbook, tablets (~1000px): Similar layout but smaller margins, > smaller plots. Coincidentally, it is about the same width of Maciej's > laptop screen ;-) > - Smartphones: iPhones and Androids in landscape mode should be now > able to visit the site without problems. > > Though it can be tested just by resizing the browser window, it would > be great to get some feedback of anyone actually using the above > mentioned devices! > > > Performance improvements > ------------------------ > > Exarkun made the timeline view faster by pre-fetching all related > data. The timeline grid took 1.46 seconds to load, and a single plot > 0.45 seconds. They now need 1.26 and 0.26 seconds respectively. A big > improvement! > > There is yet another improvement waiting in the wings, but we still > need to figure out some regression it introduces. > > The Changes view is also much faster (from 480ms to 210ms), because it > now uses the cached table stored in the corresponding report (only > when viewing with the default trend=10 ). > > > A last detail > --------------- > > Another change you may notice is that the default checked executables > in the Comparison view are now configurable (I didn't forget Maciej! > :). So they are not all selected per default any more, so the plot > shouldn't look any longer like the Flying Spaghetti Monster ;-) > > I'll leave it there. I hope you enjoy the changes! > > Miquel > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Thu Jan 20 23:11:28 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 20 Jan 2011 23:11:28 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: Message-ID: <4D38B310.304@gmail.com> Hi Miquel On 20/01/11 22:00, Miquel Torres wrote: > Hi all, > > I want to announce the release of Codespeed 0.7, the version which is > now powering speed.pypy.org wow, that's very nice, thank you once more :-). From my side, there is only one feature that I miss a lot, which is the possibility to benchmark a branch. IIRC, at the moment if you just run benchmarks on a branch, they are appended to the list of results, which can be a bit confusing especially if you look at them weeks later. I think that maybe it's possible to do it by (ab)using an additional environment, but probably a more builtin solution is better. Do you have any idea/opinion/suggestion on this? ciao, Anto From tobami at googlemail.com Fri Jan 21 08:49:50 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 21 Jan 2011 08:49:50 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: <4D38B310.304@gmail.com> References: <4D38B310.304@gmail.com> Message-ID: @Anto Yes, branches are a pending item that has been requested a couple of times now. The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things. So one idea would be: - Every revision has a branch associated to it (a problem though is that subversion has the same revision number for all branches, while mercurial and git not) - Branch would be a selectable parameter on the left menu box, just like executables or environments - You would be able to compare a branch using the comparison view, but another possibility is that the timeline view doesn't have branch as a radio button, but as a checkbox. That means that a plot is associated only to an executable and environment, and that you can choose to display either all branches in the same plot (trunk or master + any branch) or only the ones you select. - There are problems with this approach. In the Changes view, when you select a revision that doesn't have results for the currently selected branch, nothing would be displayed. You would need to blindly search for your results. An option is to always display all branches for a revision, but sometimes a revision will contain results for a branch, sometimes for others. Another idea would be to associate executables to a branch. This avoids the problem in the Changes view. But it is not really much different from the current solution of creating a different project for each branch. I would gladly hear other ideas. Cheers, Miquel 2011/1/20 Antonio Cuni : > Hi Miquel > > On 20/01/11 22:00, Miquel Torres wrote: >> >> Hi all, >> >> I want to announce the release of Codespeed 0.7, the version which is >> now powering speed.pypy.org > > wow, that's very nice, thank you once more :-). > > From my side, there is only one feature that I miss a lot, which is the > possibility to benchmark a branch. > > IIRC, at the moment if you just run benchmarks on a branch, they are > appended to the list of results, which can be a bit confusing especially if > you look at them weeks later. > > I think that maybe it's possible to do it by (ab)using an additional > environment, but probably a more builtin solution is better. Do you have any > idea/opinion/suggestion on this? > > ciao, > Anto > From p.giarrusso at gmail.com Fri Jan 21 11:24:17 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Fri, 21 Jan 2011 11:24:17 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> Message-ID: Hi Miquel! > I would gladly hear other ideas. Here are my two cents! On Fri, Jan 21, 2011 at 08:49, Miquel Torres wrote: > So one idea would be: > - Every revision has a branch associated to it (a problem though is > that subversion has the same revision number for all branches, while > mercurial and git not) This suggests (to me) that a revision would be associated to a specific branch, but then I don't get expressions like "all the branches for a revision". Maybe you mean that you would replace revisions with pairs (Revision, Branch)? [...] > - There are problems with this approach. In the Changes view, when you > select a revision that doesn't have results for the currently selected > branch, nothing would be displayed. You're referring to an SVN context, right? Otherwise I can't make sense of it. I guess you can use, on other branches, the "closest" revision available, after defining a linear order on revisions across branches. You have that order in SVN; since you don't have that in Git, why don't you just use the _date_ of a revision as the ordering key? [For the data structures to implement this, I've seen even seen interfaces to tree-based dictionaries allowing navigating to the key-predecessor and key-successor, which might be especially handy here (such as NavigableMap in Java, dunno about Python).] Of course, that's suitable just as default, so you need to be able to navigate between revisions independently for each branch. Please note that with complex merge histories, using the date does not necessarily make sense - so that at least Git tools provide many linearization algorithms. The problem is that a commit might be made on a certain date in some repository, but be merged on mainline much later, especially when many developers are not committers themselves; one solution is to consider only revisions which were committed directly on the branch or merged shortly after. I also would believe this use case does not appear for PyPy. See help about git log --topo-order and --date-order (such docs used to be more complete, though). > You would need to blindly search > for your results. An option is to always display all branches for a > revision, but sometimes a revision will contain results for a branch, > sometimes for others. Outside of the SVN context, I'm not sure I get what you mean here, because revisions in different branches are not necessarily comparable. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From eva_maia at sapo.pt Fri Jan 21 16:03:55 2011 From: eva_maia at sapo.pt (Eva Maia) Date: Fri, 21 Jan 2011 15:03:55 +0000 Subject: [pypy-dev] rPython Message-ID: <4D39A05B.60209@sapo.pt> Hi, It is possible through the PyPy toolchain check if a program is a rPython program? I know the tool Pylint (RPylint) gives an idea, but it seems to me that this tool gives some errors that result from being a little outdated. Thanks, Eva Maia. From fijall at gmail.com Fri Jan 21 16:15:40 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 21 Jan 2011 17:15:40 +0200 Subject: [pypy-dev] rPython In-Reply-To: <4D39A05B.60209@sapo.pt> References: <4D39A05B.60209@sapo.pt> Message-ID: On Fri, Jan 21, 2011 at 5:03 PM, Eva Maia wrote: > Hi, > > It is possible through the PyPy toolchain check if a program is a > rPython program? > > I know the tool Pylint (RPylint) gives an idea, but it seems to me that > this tool gives some errors that result from being a little outdated. > > Thanks, > > Eva Maia. > _______________________________________________ Hi. This tool is completely broken. Primary reason is that this tool operates on the level of source code, while RPython operates on live python objects and that's a huge difference. That said, I don't think there is any other way of finding what's RPython without compiling it. In fact, RPython is defined as a thing that can be accepted by our translation toolchain :-) PS. Why you want to write RPython in the first place? Cheers, fijal From tobami at googlemail.com Fri Jan 21 16:24:33 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 21 Jan 2011 16:24:33 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> Message-ID: Hi Paolo, >> - There are problems with this approach. In the Changes view, when you >> select a revision that doesn't have results for the currently selected >> branch, nothing would be displayed. > > You're referring to an SVN context, right? Otherwise I can't make sense of it. The opposite. For subversion, revisions mark the whole repository, so a revision could have results for both trunk and a branch. That is not the case in mercurial. Regarding my explanation, I understand it is a bit difficult to visualize this UI issues without images. >> You would need to blindly search >> for your results. An option is to always display all branches for a >> revision, but sometimes a revision will contain results for a branch, >> sometimes for others. > Outside of the SVN context, I'm not sure I get what you mean here, > because revisions in different branches are not necessarily > comparable. Same as above. So the differences between version control systems really suggest that you cannot add a branch dimension to a revision. More like a result is not only associated to a project, revision, and executable, but also to a branch... 2011/1/21 Paolo Giarrusso : > Hi Miquel! > >> I would gladly hear other ideas. > Here are my two cents! > > On Fri, Jan 21, 2011 at 08:49, Miquel Torres wrote: >> So one idea would be: >> - Every revision has a branch associated to it (a problem though is >> that subversion has the same revision number for all branches, while >> mercurial and git not) > This suggests (to me) that a revision would be associated to a > specific branch, but then I don't get expressions like "all the > branches for a revision". Maybe you mean that you would replace > revisions with pairs (Revision, Branch)? > [...] >> - There are problems with this approach. In the Changes view, when you >> select a revision that doesn't have results for the currently selected >> branch, nothing would be displayed. > > You're referring to an SVN context, right? Otherwise I can't make sense of it. > > I guess you can use, on other branches, the "closest" revision > available, after defining a linear order on revisions across branches. > You have that order in SVN; since you don't have that in Git, why > don't you just use the _date_ of a revision as the ordering key? > [For the data structures to implement this, I've seen even seen > interfaces to tree-based dictionaries allowing navigating to the > key-predecessor and key-successor, which might be especially handy > here (such as NavigableMap in Java, dunno about Python).] > Of course, that's suitable just as default, so you need to be able to > navigate between revisions independently for each branch. > > Please note that with complex merge histories, using the date does not > necessarily make sense - so that at least Git tools provide many > linearization algorithms. The problem is that a commit might be made > on a certain date in some repository, but be merged on mainline much > later, especially when many developers are not committers themselves; > one solution is to consider only revisions which were committed > directly on the branch or merged shortly after. > I also would believe this use case does not appear for PyPy. > See help about git log --topo-order and --date-order (such docs used > to be more complete, though). > >> You would need to blindly search >> for your results. An option is to always display all branches for a >> revision, but sometimes a revision will contain results for a branch, >> sometimes for others. > Outside of the SVN context, I'm not sure I get what you mean here, > because revisions in different branches are not necessarily > comparable. > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > From dimaqq at gmail.com Fri Jan 21 19:07:39 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 21 Jan 2011 11:07:39 -0700 Subject: [pypy-dev] rPython In-Reply-To: <4D39A05B.60209@sapo.pt> References: <4D39A05B.60209@sapo.pt> Message-ID: Hi, Eva I assume you are asking if there is a program is a valid rpython program, right? Since rpython is a subset of python, you can check if it is a valid python program and then verify the restrictions imposed by rpython. "RPython: a Step Towards Reconciling Dynamically and Statically Typed OO Languages" lists these. PyPy impementation of rpython appears to check these constraints at "initialization" time, which is in a sense, python runtime. The result is, I think, to try and see if it compiles. Come to think of it, vaildity and correctness are blurred in dynamic languages. Syntax is checked, but then Imports and asserts can affect if all of the program is even considered. Alternatively if you want to quickly distinguish practical rpython code from python, I guess you could look for clues such as what types are used, what modules imported, etc. Dima On 21 January 2011 08:03, Eva Maia wrote: > Hi, > > It is possible through the PyPy toolchain check if a program is a > rPython program? > > I know the tool Pylint (RPylint) gives an idea, but it seems to me that > this tool gives some errors that result from being a little outdated. > > Thanks, > > Eva Maia. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Fri Jan 21 19:14:28 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 21 Jan 2011 19:14:28 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> Message-ID: <4D39CD04.3050104@gmail.com> On 21/01/11 08:49, Miquel Torres wrote: > @Anto > > Yes, branches are a pending item that has been requested a couple of times now. yes, I think most of the requests has been by me :) > The current solution is actually not to abuse an environment like you > say, but to create a new project for a branch. That way it gets > cleanly separated, and in a way branches are like different projects. > But it is of course not optimal. Technically it is very easy to come > up with several solutions (add a branch dimension, for example), but > interface-wise it is not easy to find something that doesn't clutter > things. Uhm, I don't think that using a different project is a good idea. For branches, we are usually not much interested in the branch history, but in the comparison with trunk (ideally, with trunk at the point we created the branch, or at the point of the last merge from trunk). As for visualize changes, I think that we don't need anything fancy, for example it would be already immensely useful to have the possibility of displaying the Changes page in a way that it compares the performances of the branch against trunk. ciao, Anto From exarkun at twistedmatrix.com Fri Jan 21 19:34:11 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 21 Jan 2011 18:34:11 -0000 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: <4D39CD04.3050104@gmail.com> References: <4D38B310.304@gmail.com> <4D39CD04.3050104@gmail.com> Message-ID: <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> On 06:14 pm, anto.cuni at gmail.com wrote: >On 21/01/11 08:49, Miquel Torres wrote: >>@Anto >> >>Yes, branches are a pending item that has been requested a couple of >>times now. > >yes, I think most of the requests has been by me :) >>The current solution is actually not to abuse an environment like you >>say, but to create a new project for a branch. That way it gets >>cleanly separated, and in a way branches are like different projects. >>But it is of course not optimal. Technically it is very easy to come >>up with several solutions (add a branch dimension, for example), but >>interface-wise it is not easy to find something that doesn't clutter >>things. > >Uhm, I don't think that using a different project is a good idea. For >branches, we are usually not much interested in the branch history, but >in the >comparison with trunk (ideally, with trunk at the point we created the >branch, >or at the point of the last merge from trunk). > >As for visualize changes, I think that we don't need anything fancy, >for >example it would be already immensely useful to have the possibility of >displaying the Changes page in a way that it compares the performances >of the >branch against trunk. How about a "Branches" checkbox (per project? per executable? per graph? One of those maybe). When it's checked, branch results within the revision horizon (last 10, 50, 200, etc) get plotted on the same graph as the trunk data is plotted. Each branch could be a different color, perhaps (but would at least have hover info telling you what it is). This implies adding a branch column to the results table (or is it the revisions table?). Maybe that's just the obvious way to do it and everyone else already thought of and discarded it already, though. Actually, in general I'd like a way to plot more things on one graph. So maybe this is just a special case of that. Jean-Paul From samuele.pedroni at gmail.com Fri Jan 21 21:29:19 2011 From: samuele.pedroni at gmail.com (Samuele Pedroni) Date: Fri, 21 Jan 2011 21:29:19 +0100 Subject: [pypy-dev] Branches in the mercurial repository In-Reply-To: <4D394D85.2010106@openend.se> References: <201101201616.46698.jacob@openend.se> <4D394D85.2010106@openend.se> Message-ID: On Fri, Jan 21, 2011 at 10:10 AM, Bea During wrote: >> On Thu, Jan 20, 2011 at 10:33 AM, Samuele Pedroni >> > wrote: >> >> ? ?> Samuele ? ? ? ? ? ? ?ctypes-stable ? ? ? ? ? ?Fri Apr 11 >> ? ?15:44:57 2008 >> ? ?> Samuele ? ? ? ? ? ? ?run-django ? ? ? ? ? ? ? Sat Jul 12 >> ? ?09:46:07 2008 ... >> closed just means that they are no longer being used (and won't be shown >> in the default list of branches), I tihnk those are safe to close. >> > +1. > closed Samuele From tobami at googlemail.com Sun Jan 23 20:37:39 2011 From: tobami at googlemail.com (Miquel Torres) Date: Sun, 23 Jan 2011 20:37:39 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> References: <4D38B310.304@gmail.com> <4D39CD04.3050104@gmail.com> <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> Message-ID: It doesn't sound like a bad idea. But how would you save the branch data? 2011/1/21 : > On 06:14 pm, anto.cuni at gmail.com wrote: >>On 21/01/11 08:49, Miquel Torres wrote: >>>@Anto >>> >>>Yes, branches are a pending item that has been requested a couple of >>>times now. >> >>yes, I think most of the requests has been by me :) >>>The current solution is actually not to abuse an environment like you >>>say, but to create a new project for a branch. That way it gets >>>cleanly separated, and in a way branches are like different projects. >>>But it is of course not optimal. Technically it is very easy to come >>>up with several solutions (add a branch dimension, for example), but >>>interface-wise it is not easy to find something that doesn't clutter >>>things. >> >>Uhm, I don't think that using a different project is a good idea. For >>branches, we are usually not much interested in the branch history, but >>in the >>comparison with trunk (ideally, with trunk at the point we created the >>branch, >>or at the point of the last merge from trunk). >> >>As for visualize changes, I think that we don't need anything fancy, >>for >>example it would be already immensely useful to have the possibility of >>displaying the Changes page in a way that it compares the performances >>of the >>branch against trunk. > > How about a "Branches" checkbox (per project? ?per executable? ?per > graph? ?One of those maybe). ?When it's checked, branch results within > the revision horizon (last 10, 50, 200, etc) get plotted on the same > graph as the trunk data is plotted. ?Each branch could be a different > color, perhaps (but would at least have hover info telling you what it > is). > > This implies adding a branch column to the results table (or is it the > revisions table?). > > Maybe that's just the obvious way to do it and everyone else already > thought of and discarded it already, though. > > Actually, in general I'd like a way to plot more things on one graph. > So maybe this is just a special case of that. > > Jean-Paul > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From exarkun at twistedmatrix.com Mon Jan 24 00:37:27 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 23 Jan 2011 23:37:27 -0000 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> <4D39CD04.3050104@gmail.com> <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> Message-ID: <20110123233727.1699.1400508058.divmod.xquotient.222@localhost.localdomain> On 07:37 pm, tobami at googlemail.com wrote: >It doesn't sound like a bad idea. But how would you save the branch >data? What do you need beyond an extra column in the revisions table? Everything else would be the same, except when you upload data, include the branch it is for, and when you query for data, limit yourself to trunk/default/whatever unless you know you want more. Jean-Paul > >2011/1/21 : >>On 06:14 pm, anto.cuni at gmail.com wrote: >>>On 21/01/11 08:49, Miquel Torres wrote: >>>>@Anto >>>> >>>>Yes, branches are a pending item that has been requested a couple of >>>>times now. >>> >>>yes, I think most of the requests has been by me :) >>>>The current solution is actually not to abuse an environment like >>>>you >>>>say, but to create a new project for a branch. That way it gets >>>>cleanly separated, and in a way branches are like different >>>>projects. >>>>But it is of course not optimal. Technically it is very easy to come >>>>up with several solutions (add a branch dimension, for example), but >>>>interface-wise it is not easy to find something that doesn't clutter >>>>things. >>> >>>Uhm, I don't think that using a different project is a good idea. For >>>branches, we are usually not much interested in the branch history, >>>but >>>in the >>>comparison with trunk (ideally, with trunk at the point we created >>>the >>>branch, >>>or at the point of the last merge from trunk). >>> >>>As for visualize changes, I think that we don't need anything fancy, >>>for >>>example it would be already immensely useful to have the possibility >>>of >>>displaying the Changes page in a way that it compares the >>>performances >>>of the >>>branch against trunk. >> >>How about a "Branches" checkbox (per project? ?per executable? ?per >>graph? ?One of those maybe). ?When it's checked, branch results within >>the revision horizon (last 10, 50, 200, etc) get plotted on the same >>graph as the trunk data is plotted. ?Each branch could be a different >>color, perhaps (but would at least have hover info telling you what it >>is). >> >>This implies adding a branch column to the results table (or is it the >>revisions table?). >> >>Maybe that's just the obvious way to do it and everyone else already >>thought of and discarded it already, though. >> >>Actually, in general I'd like a way to plot more things on one graph. >>So maybe this is just a special case of that. >> >>Jean-Paul >>_______________________________________________ >>pypy-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/pypy-dev > From Arnd.Rechenburg at tomtom.com Mon Jan 24 09:38:52 2011 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Mon, 24 Jan 2011 09:38:52 +0100 Subject: [pypy-dev] PyObject_AsCharBuffer: compiler warning Message-ID: Hi, When I try to use the function PyObject_AsCharBuffer I get the following compiler warning: warning: passing argument 2 of 'PyObject_AsCharBuffer' from incompatible pointer type By using Python it works without problems. In Python: int PyObject_AsCharBuffer(PyObject *obj, const char **buffer, Py_ssize_t *buffer_len) Is there something different in pypy? Regards, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Mon Jan 24 09:41:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Jan 2011 10:41:49 +0200 Subject: [pypy-dev] PyObject_AsCharBuffer: compiler warning In-Reply-To: References: Message-ID: On Mon, Jan 24, 2011 at 10:38 AM, Arnd Rechenburg wrote: > Hi, > > > > When I try to use the function PyObject_AsCharBuffer I get the following > compiler warning: > > warning: passing argument 2 of 'PyObject_AsCharBuffer' from incompatible > pointer type > > > > By using Python it works without problems. > > In Python: > > int PyObject_AsCharBuffer(PyObject *obj, const char **buffer, Py_ssize_t > *buffer_len) > > > > Is there something different in pypy? > > We don't have const char**, we use char** instead. It's a small deficiency of how we do stuff now, probably fixable in the future, but don't worry too much, it works the same way. Cheers, fijal From fijall at gmail.com Mon Jan 24 20:01:25 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Jan 2011 21:01:25 +0200 Subject: [pypy-dev] Fwd: [issue10994] implementation details in sys module In-Reply-To: <1295878409.3679.12.camel@localhost.localdomain> References: <1295877929.5.0.527131313453.issue10994@psf.upfronthosting.co.za> <1295878409.3679.12.camel@localhost.localdomain> Message-ID: I think this explanation what sys.getsizeof does is interesting. We might want to have it. ---------- Forwarded message ---------- From: Antoine Pitrou Date: Mon, Jan 24, 2011 at 4:13 PM Subject: [issue10994] implementation details in sys module To: fijall at gmail.com Antoine Pitrou added the comment: > I suppose wrt getsizeof it's more of "if you provide us with a > reasonable expectations, we can implement this" other than anything > else. The expectation is that it returns the memory footprint of the given object, and only it (not taking into account sharing, caching, dependencies or anything else). For example, an instance will not count its attribute __dict__. But a str object will count its object header plus the string payload, if the payload is private. Of course, you are free to tweak these semantics for the PyPy implementation. ---------- _______________________________________ Python tracker _______________________________________ From tvoglou at gmail.com Mon Jan 31 22:27:53 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Mon, 31 Jan 2011 23:27:53 +0200 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit Message-ID: Hello All, I am trying to translate and compile pypy using Visual Studio 2010 to Win64a and I am getting the following issue with ssl. [translation:WARNING] The module '_ssl' is disabled [translation:WARNING] because importing pypy.module._ssl.interp_ssl raised CompilationError [translation:WARNING] After that the translation operation stops and the debugger kicks in. The strange part is that when I execute cl/link from a command prompt the file is compiled and linked without issues. I've compiled OpenSSL with 64bit support removing bufferoverflowu.lib which is not needed in vs2010. Regards Tasos From amauryfa at gmail.com Mon Jan 31 22:36:30 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 31 Jan 2011 22:36:30 +0100 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: Hi, 2011/1/31 Tasos Vogiatzoglou : > Hello All, > > I am trying to translate and compile pypy using Visual Studio 2010 to > Win64a Wow, you are the first one! But I don't think it will work: sadly in PyPy there is the implicit assumption that a pointer can be stored in a C "long", which is wrong on win64. > and I am getting the following issue with ssl. > [translation:WARNING] The module '_ssl' is disabled > [translation:WARNING] because importing pypy.module._ssl.interp_ssl > raised CompilationError > [translation:WARNING] > > After that the translation operation stops and the debugger kicks in. But these are *warnings*. Which error do you get exactly? do you have a (red) traceback? -- Amaury Forgeot d'Arc From tvoglou at gmail.com Mon Jan 31 22:39:25 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Mon, 31 Jan 2011 23:39:25 +0200 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: Hello Amaury, Yes. It's the same compiler error : [platform:ERROR] platcheck_49.obj : error LNK2019: unresolved external symbol _SSLeay_version referenced in function _dump_section_SSLEAY_VERSION [platform:ERROR] c:\users\flatline\appdata\local\temp\usession-default-56\platcheck_49.exe : fatal error LNK1120: 1 unresolved externals [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 268, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 831, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 229, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 240, in get_entry_point [translation:ERROR] space = make_objspace(config) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\tool\option.py", line 48, in make_objspace [translation:ERROR] space = Space(config) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 280, in __init__ [translation:ERROR] self.initialize() [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\objspace\std\objspace.py", line 78, in initialize [translation:ERROR] self.make_builtins() [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 490, in make_builtins [translation:ERROR] self.install_mixedmodule(mixedname, installed_builtin_modules) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 529, in install_mixedmodule [translation:ERROR] modname = self.setbuiltinmodule(mixedname) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 370, in setbuiltinmodule [translation:ERROR] None, None, ["Module"]).Module [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\__init__.py", line 2, in [translation:ERROR] from pypy.module._hashlib.interp_hashlib import algorithms [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py", line 9, in [translation:ERROR] from pypy.rlib import ropenssl [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\rlib\ropenssl.py", line 76, in [translation:ERROR] for k, v in rffi_platform.configure(CConfig).items(): [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\rpython\tool\rffi_platform.py", line 201, in configure [translation:ERROR] infolist = list(run_example_code(writer.path, eci)) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\rpython\tool\rffi_platform.py", line 685, in run_example_code [translation:ERROR] output = build_executable_cache(files, eci) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\tool\gcc_cache.py", line 27, in build_executable_cache [translation:ERROR] result = platform.execute(platform.compile(c_files, eci) ) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 52, in compile [translation:ERROR] return self._finish_linking(ofiles, eci, outputfilename, standalone) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 194, in _finish_linking [translation:ERROR] return self._link(cc_link, ofiles, largs, standalone, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 176, in _link [translation:ERROR] self._execute_c_compiler(self.link, args, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 118, in _execute_c_compiler [translation:ERROR] self._handle_error(returncode, stderr, stdout, outname) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 199, in _handle_error [translation:ERROR] raise CompilationError(stdout, stderr) [translation:ERROR] CompilationError: Regards Tasos Vogiatzoglou On Mon, Jan 31, 2011 at 11:36 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/1/31 Tasos Vogiatzoglou : >> Hello All, >> >> I am trying to translate and compile pypy using Visual Studio 2010 to >> Win64a > > Wow, you are the first one! But I don't think it will work: sadly in > PyPy there is the implicit assumption that a pointer can be stored in > a C "long", which is wrong on win64. > >> ?and I am getting the following issue with ssl. >> [translation:WARNING] The module '_ssl' is disabled >> [translation:WARNING] because importing pypy.module._ssl.interp_ssl >> raised CompilationError >> [translation:WARNING] >> >> After that the translation operation stops and the debugger kicks in. > > But these are *warnings*. Which error do you get exactly? do you have > a (red) traceback? > > -- > Amaury Forgeot d'Arc > From amauryfa at gmail.com Mon Jan 31 22:44:31 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 31 Jan 2011 22:44:31 +0100 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: 2011/1/31 Tasos Vogiatzoglou : > Hello Amaury, > > Yes. It's the same compiler error : > [translation:ERROR] ? ?File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py", Ah, indeed the _hashlib module should be disabled when openssl cannot be processed. I'll try to fix it, in the meantime I suggest you to skip this module: python translate.py targetpypystandalone --withoutmod-_hashlib -- Amaury Forgeot d'Arc From tvoglou at gmail.com Mon Jan 31 22:48:57 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Mon, 31 Jan 2011 23:48:57 +0200 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: Amaury, Isn't there a way to fix the SSL issue? I was looking checking for name decoration issues or call-conv issues but there seems none. Is there any guess why it fails from the translation procedure but works when I execute the commands in cmd prompt ? (I also tried disabling the env processing that is happening, leaving POpen to get the default one but nothing :( ) Thanks, Tasos On Mon, Jan 31, 2011 at 11:44 PM, Amaury Forgeot d'Arc wrote: > 2011/1/31 Tasos Vogiatzoglou : >> Hello Amaury, >> >> Yes. It's the same compiler error : > >> [translation:ERROR] ? ?File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py", > > Ah, indeed the _hashlib module should be disabled when openssl cannot > be processed. > I'll try to fix it, in the meantime I suggest you to skip this module: > > python translate.py targetpypystandalone --withoutmod-_hashlib > > > -- > Amaury Forgeot d'Arc >