From tismer at stackless.com Mon May 1 01:54:05 2006 From: tismer at stackless.com (Christian Tismer) Date: Sun, 30 Apr 2006 16:54:05 -0700 Subject: [pypy-dev] missed the meeting In-Reply-To: <20060429191527.GA15749@code0.codespeak.net> References: <44525AA8.5020306@stackless.com> <20060429191527.GA15749@code0.codespeak.net> Message-ID: <44554E1D.4010007@stackless.com> Armin Rigo wrote: > Hi Christian, > > On Fri, Apr 28, 2006 at 11:10:48AM -0700, Christian Tismer wrote: >> for some reason I was sure that we cancelled this week's >> sync meeting. This was absolutely unintentional, >> very sorry about that! > > It was not a normal sync meeting. I called for a stackless meeting on > Friday, as we decided on Tuesday. I wouldn't hide the fact that we were > a bit disappointed not to see you there, and also not to see any docs > checked in, countrary to what you promized... I can imagine. Although I do admit that I tend to be a sloppy and lazy guy, this time I was really hit by the ongoings with my son, which happened right after the Tuesday meeting. It drove me crazy, I could neither sleep nor work nor even think for the rest of the week, and it made me a very unhappy man. I think I grew a lot of gray hair. My son has left my house, stopped his school six weeks before the final exams, which he would have made (I asked the teachers), in order to live near to his girlfriend and work for some company, without Abitur and a very limited future. This is such a slap into my face that I have a hard time recovering from "loosing" my son. > We talked a bit about > ways to go forward but without deciding anything yet, given that we are > not sure what exactly you have in mind about the hardest problems we > foresee. Now Michael is on holidays, and people are travelling back > from the sprint, so we'll try again maybe on Monday... Monday isn't perfect, since I'm traveling back in the afternoon and will be very busy, but I will do my best to be on pypy-sync? at 5pm your time. The thing we need to discuss is about the "essence of Stackless", and it is crucial to get this well understood and done right. I will force myself right now to finish the design document and give a clear picture about how I think the Stackless part of PyPy should be finished with the minimum impact on the current implementation. all the best - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon May 1 10:21:43 2006 From: tismer at stackless.com (Christian Tismer) Date: Mon, 01 May 2006 01:21:43 -0700 Subject: [pypy-dev] The Essentials of Stackless Message-ID: <4455C517.3080004@stackless.com> Dear friends, I've finally come up with a rough document, which for my vision of things says almost everything what needs to be said about Stackless Python and how to implement it for PyPy, especially regarding how to implement the thread pickling. It may be incomplete and not well-designed, it is more like a brain-dump after a week of more or less thinking, pretending to be a structured document. In the end, as an addition, I come to an interesting conclusion about life-ness analysis and GC transform, that I really did not expect beforehand. Eventually, I'm beginning to understand what I did when I wrote Stackless Python. please have a look and comment about http://codespeak.net:/svn/pypy/dist/pypy/doc/discussion/howtoimplementpickling.txt all the best -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From seojiwon at gmail.com Tue May 2 07:33:47 2006 From: seojiwon at gmail.com (Jiwon Seo) Date: Mon, 1 May 2006 22:33:47 -0700 Subject: [pypy-dev] Error on translation? Message-ID: Hi all, I am trying to run translated versions of targetnopstandalone.py, targetrpystonedalone.py (and eventually pypy-c) on a simulator. However, even before that, the translated versions on real machine (so, not on a simulator) crash. (Machine spec and options for translate.py are at the end) For instance, translated version of targetrpystonedalone.py produces segmentation fault when I ran it. The stack when segfault occurs looks like following: _RPyListOfString_SetItem(list, i, s) in main.h pypy_g__RPyListOfString_SetItem__listPtr_Signed_rpy_str(struct pypy_list0 *l_l_1, long l_index_0, struct pypy_rpy_string0 *l_newstring_0) in implement.c pypy_g_ll_setitem_nonneg__dum_nocheckConst_listPtr_Sign(struct pypy_list0 *l_l_2, long l_index_1, struct pypy_rpy_string0 *l_newitem_2) in implement.c and inside the last function, segmentation fault occurs at l_v10 = l_l_2->l_items; because l_l_2 is not allocated. (actually, I don't see any place that l_l_2 is allocated before that line) I'm runnint the translated executable on ppc linux: (uname -a shows following: Linux rundmc.stanford.edu 2.6.15-1.2054_FC5 #1 SMP Tue Mar 14 15:57:06 EST 2006 ppc64 ppc64 ppc64 GNU/Linux) and I ran translate.py with --text --no-compile --source --gc=ref options, and then mv /tmp/usession-x/testing_1/ to a new place, then compiled it. Ah, and I'm using current version of pypy. Any idea? -Jiwon From sanxiyn at gmail.com Tue May 2 08:49:00 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Tue, 2 May 2006 15:49:00 +0900 Subject: [pypy-dev] oopspec Message-ID: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> Is there some documentation on oopspec attribute? How one may use it in the backend? Seo Sanghyeon From sanxiyn at gmail.com Tue May 2 09:02:46 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Tue, 2 May 2006 16:02:46 +0900 Subject: [pypy-dev] Signature PEP Message-ID: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> Hi, PyPy people, What do you think about this Signature PEP? http://mail.python.org/pipermail/python-3000/2006-April/001249.html PyPy has something similar as pypy.interpreter.pycode.cpython_code_signature, and I think it's definitely a good idea to have some better interface than code.co_* and CO_FLAGS. http://codespeak.net/svn/pypy/dist/pypy/interpreter/pycode.py We also have more extensive interface in pypy.annotation.description: http://codespeak.net/svn/pypy/dist/pypy/annotation/description.py There's FunctionDesc, ClassDesc, and MethodDesc there. Though they are quite PyPy-specific. Seo Sanghyeon From pedronis at strakt.com Tue May 2 10:09:52 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Tue, 02 May 2006 10:09:52 +0200 Subject: [pypy-dev] Google summer of code is open student for submissions Message-ID: <445713D0.7060906@strakt.com> Deadline is the 8th of May. PyPy related projects may be submitted under the PSF umbrella. A list of possible PyPy project ideas can be found at: http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas.txt regards. From fabien-ml at x-phuture.com Tue May 2 11:57:10 2006 From: fabien-ml at x-phuture.com (Fabien Schwob) Date: Tue, 02 May 2006 11:57:10 +0200 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <445713D0.7060906@strakt.com> Message-ID: <20060502095710.45DA67EC9B@postix.sdv.fr> > A list of possible PyPy project ideas can be found at: > > http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas > .txt I'm really interested in the following project : * Write an interpreter for **another dynamic language** in the PyPy framework. For example, a Javascript interpreter would be suitable. Ruby too (though the latter is probably more than two months of work). Or Scheme, or... etc. And I would like to have more information about it if possible. I would like to check if I understand it. So the goal would be to parse the Javascript instructions and to create an AST tree from it ? So, it would be possible to write Javascript code that can be after executed on the PyPy backends (CLI, C, LLVM, etc...). Thanks in advance. From anto.cuni at gmail.com Tue May 2 12:01:27 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 02 May 2006 12:01:27 +0200 Subject: [pypy-dev] oopspec In-Reply-To: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> Message-ID: <44572DF7.90900@gmail.com> Hi Seo, Sanghyeon Seo wrote: > Is there some documentation on oopspec attribute? How one may use it > in the backend? I think there are no docs about the oopspec attribute. Armin wrote these lines some time ago to respond to the same question: """ The oopspec string tells what is the "abstract" list operation that this particular ll_*() function implement. For example: def ll_prepend(l, newitem): ... ll_prepend.oopspec = 'list.insert(l, 0, newitem)' means that ll_prepend() is equivalent to an insert with the index set to zero. In the stirng, the pseudo-arguments between the ( ) are either real argument names of the ll_ function, or constants. So for example, if a backend has got its own way to implement the insert() calls in general, it could figure out from the oopspec that the ll_prepend() helper can be replaced by a custom stub invoking the backend's own version of insertion with an index of 0. That's essentially what the JIT does -- see handle_highlevel_operation() in jit/hintannotator/model.py. """ The CLI backend uses the oopspec attribute for replacing calls to selected low-level helpers with native builtin methods; the code is still very experimental since it doesn't parse the argument line: it simply forwards the call using the first parameter as the target object and subsequent parameters as method's arguments. By now the only recognized oopspec is 'list.append' (i.e., ll_append) that is translated to the 'Add' method. If you want to look at my code see translator/cli/oopspec.py and the _Call.render method in translator/cli/metavm.py; at the moment I put these files in the cli/ directory, but they are general enough to be shared among multiple backends (metavm would be useful only for backends emitting bytecode, so not for gencl nor gensqueak, I guess). The rationale behind this is that this way a backend can quickly gain full list support by simply supply basic operations such as ll_getitem_fast & Co; then each backend can choose what operation to optimize based on their knowledge of the target system. Btw, I have a doubt about oopspec, too: Armin told that the 'oopspec' specifies the "abstract" operation that each ll_* helper implements; does this abstract operation have to be one of standard list methods or can I add new operations? I was thinking to add a 'll_remove_range' or similar to be used by other helpers such as delslice and company; this way backends can replace ll_remove_range with their almost-surely-present equivalent without having to care about python-specific logic such as slices or so. ciao Anto From seojiwon at gmail.com Tue May 2 19:33:19 2006 From: seojiwon at gmail.com (Jiwon Seo) Date: Tue, 2 May 2006 10:33:19 -0700 Subject: [pypy-dev] Re: Error on translation? In-Reply-To: References: Message-ID: Um, I was using wrong main.h. Thank Armin for the help. -Jiwon On 5/1/06, Jiwon Seo wrote: > Hi all, > > I am trying to run translated versions of targetnopstandalone.py, > targetrpystonedalone.py (and eventually pypy-c) on a simulator. > However, even before that, the translated versions on real machine > (so, not on a simulator) crash. (Machine spec and options for > translate.py are at the end) > > For instance, translated version of targetrpystonedalone.py produces > segmentation fault when I ran it. The stack when segfault occurs > looks like following: > > _RPyListOfString_SetItem(list, i, s) in main.h > > pypy_g__RPyListOfString_SetItem__listPtr_Signed_rpy_str(struct > pypy_list0 *l_l_1, long l_index_0, struct pypy_rpy_string0 > *l_newstring_0) in implement.c > > pypy_g_ll_setitem_nonneg__dum_nocheckConst_listPtr_Sign(struct > pypy_list0 *l_l_2, long l_index_1, struct pypy_rpy_string0 > *l_newitem_2) in implement.c > > and inside the last function, segmentation fault occurs at > l_v10 = l_l_2->l_items; > > because l_l_2 is not allocated. (actually, I don't see any place that > l_l_2 is allocated before that line) > > I'm runnint the translated executable on ppc linux: > (uname -a shows following: Linux rundmc.stanford.edu 2.6.15-1.2054_FC5 > #1 SMP Tue Mar 14 15:57:06 EST 2006 ppc64 ppc64 ppc64 GNU/Linux) > > and I ran translate.py with --text --no-compile --source --gc=ref > options, and then mv /tmp/usession-x/testing_1/ to a new place, then > compiled it. Ah, and I'm using current version of pypy. > > Any idea? > > -Jiwon > From arigo at tunes.org Wed May 3 00:06:24 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 3 May 2006 00:06:24 +0200 Subject: [pypy-dev] oopspec In-Reply-To: <44572DF7.90900@gmail.com> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> <44572DF7.90900@gmail.com> Message-ID: <20060502220624.GA7698@code0.codespeak.net> Hi Antonio, On Tue, May 02, 2006 at 12:01:27PM +0200, Antonio Cuni wrote: > Btw, I have a doubt about oopspec, too: Armin told that the 'oopspec' > specifies the "abstract" operation that each ll_* helper implements; > does this abstract operation have to be one of standard list methods or > can I add new operations? I was thinking to add a 'll_remove_range' or > similar to be used by other helpers such as delslice and company; this > way backends can replace ll_remove_range with their > almost-surely-present equivalent without having to care about > python-specific logic such as slices or so. It's quite open; every piece of code using oopspec should be prepared to see names that it doesn't know about, and ignore them. So feel free to add new names. As a guideline, let's stick as far as possible to the Python name for the method or for the __xxx__ special method name (with __ removed): 'remove_range' looks like it could be called 'delslice' in oopspec. A bientot, Armin From arigo at tunes.org Wed May 3 00:11:47 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 3 May 2006 00:11:47 +0200 Subject: [pypy-dev] Signature PEP In-Reply-To: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> References: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> Message-ID: <20060502221146.GB7698@code0.codespeak.net> Hi Seo, On Tue, May 02, 2006 at 04:02:46PM +0900, Sanghyeon Seo wrote: > What do you think about this Signature PEP? > http://mail.python.org/pipermail/python-3000/2006-April/001249.html It's a language design issue so it's off topic :-) More seriously, sure, it would be nice to have some kind of standard. Here, and in many similar code-manipulation places in PyPy, we had to hack together our own minimal solution. A harder problem along these lines would be a better way to build pieces of code, in order to create functions from scratch. \n-joining together source lines full of %s, avoiding name clashes, and getting the indentation right, is not fun. A bientot, Armin From arigo at tunes.org Wed May 3 00:40:19 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 3 May 2006 00:40:19 +0200 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <20060502095710.45DA67EC9B@postix.sdv.fr> References: <445713D0.7060906@strakt.com> <20060502095710.45DA67EC9B@postix.sdv.fr> Message-ID: <20060502224019.GC7698@code0.codespeak.net> Hi Fabien, On Tue, May 02, 2006 at 11:57:10AM +0200, Fabien Schwob wrote: > And I would like to have more information about it if possible. I would > like to check if I understand it. So the goal would be to parse the > Javascript instructions and to create an AST tree from it ? So, it would > be possible to write Javascript code that can be after executed on the > PyPy backends (CLI, C, LLVM, etc...). The goal is to write an *interpreter*. At this point it's completely unrelated to the translation backends. Let's consider Javascript (the same argument applies to any interpreted language): to write an interpreter, some people focus on the task of compiling, say, Javascript source code to get an AST; this is only part of the job, and it should be the shortest part -- ideally, finding an existing parser would be best, otherwise plugging one together with existing tools shouldn't take too long for a regular language like Javascript. It could be an AST or bytecode in any format. The core of the work is actually to build an interpreter for this AST or bytecode, because in addition to the AST or bytecode interpreter you also need to put together the object model, and build the standard types. This means basically rewriting a Standard Interpreter that interprets Javascript instead of full Python -- for what Standard Interpreter exactly means here, see http://codespeak.net/pypy/dist/pypy/doc/architecture.html . (Getting the basics working to interpret Javascript is rather easier than to interpret Python, as the JS language and object model are far simpler; nevertheless, a two-months job will probably leave many details unpolished, which is ok.) Per se, this goal is not related to the back-ends of the Translation Process of PyPy (see again the same link for "Translation Process"). So it's not about providing either a front-end nor a back-end for Javascript. The reason it's still interesting to write a JS interpreter is that the JS interpreter itself can then be turned into C code, or .NET code, or any of the platforms that the Translation Process supports or will support. In this way, writing a JS interpreter onces gives you Javascript.NET, Javascript-on-Java, etc. all at once, using the unmodified translation toolchain. This works for any program written in the PyPy framework (i.e. written in RPython); but the reason PyPy is well suited to write - more specifically - *interpreters* instead of something else, is about what interesting things we can do with the RPython program if it's an interpreter. This means e.g. turn it into a Just-In-Time compiler. Our goal (not achieved yet) is that with relatively little efforts, each of the interpreters written in the PyPy framework will be turned into JIT compilers. So we might later obtain a Javascript JIT out of the Summer of Code-sized Javascript interpreter :-) Clearly, that's outside the scope of the project, that's just motivation. A bientot, Armin. From anto.cuni at gmail.com Wed May 3 01:00:28 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 03 May 2006 01:00:28 +0200 Subject: [pypy-dev] oopspec In-Reply-To: <20060502220624.GA7698@code0.codespeak.net> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> <44572DF7.90900@gmail.com> <20060502220624.GA7698@code0.codespeak.net> Message-ID: <4457E48C.4070408@gmail.com> Hi Armin Armin Rigo wrote: > It's quite open; every piece of code using oopspec should be prepared to > see names that it doesn't know about, and ignore them. So feel free to > add new names. As a guideline, let's stick as far as possible to the > Python name for the method or for the __xxx__ special method name (with > __ removed): 'remove_range' looks like it could be called 'delslice' in > oopspec. I'm unsure about using 'delslice': it make me think that such an operation behave exactly like the corresponding python statement, but that was not my intent: in my mind the difference between 'remove_range' and 'delslice' is that the first doesn't handle negative indexes and so it is likely that the target has some built-in method that implement it natively. Btw I have not thought to it too much, so I don't know if such a refactoring is worth the pain: maybe it is simply more effective to write a delslice method in C# (or whatever language fit other backends) and forwarding the hypothetic 'delslice' to it, even it would mean that each backend has to write its own implementation if they don't get satisfaction with the default one. ciao Anto From sanxiyn at gmail.com Wed May 3 02:32:21 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Wed, 3 May 2006 09:32:21 +0900 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <20060502224019.GC7698@code0.codespeak.net> References: <445713D0.7060906@strakt.com> <20060502095710.45DA67EC9B@postix.sdv.fr> <20060502224019.GC7698@code0.codespeak.net> Message-ID: <5b0248170605021732j383ba149tffdc0b7d1dde9ef1@mail.gmail.com> 2006/5/3, Armin Rigo : > Let's consider Javascript (the same argument applies to any interpreted > language): to write an interpreter, some people focus on the task of > compiling, say, Javascript source code to get an AST; this is only part > of the job, and it should be the shortest part -- ideally, finding an > existing parser would be best, otherwise plugging one together with > existing tools shouldn't take too long for a regular language like > Javascript. Sadly, parsing Javascript is unnecessarily difficult because of optional semicolon insertion. Best to look at existing parsers. Seo Sanghyeon From nnorwitz at gmail.com Wed May 3 09:15:33 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 3 May 2006 00:15:33 -0700 Subject: [pypy-dev] Seeking students for the Summer of Code Message-ID: There is less than a week left before students must submit a final application. There are a bunch of ideas up on the wiki: http://wiki.python.org/moin/SummerOfCode/ The wiki has instructions for how to submit a proposal. There are many different areas including: core language features, libraries, and applications. This is a great opportunity to get real coding experience. Not to mention the chance to work with a nice and fun group of people. The earlier you submit an application, the more feedback you can get to improve it and increase your liklihood of getting accepted. Feel free to contact me if you have any questions. Cheers, n From arigo at tunes.org Wed May 3 09:22:20 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 3 May 2006 09:22:20 +0200 Subject: [pypy-dev] oopspec In-Reply-To: <4457E48C.4070408@gmail.com> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> <44572DF7.90900@gmail.com> <20060502220624.GA7698@code0.codespeak.net> <4457E48C.4070408@gmail.com> Message-ID: <20060503072220.GB6154@code0.codespeak.net> Hi Antonio, On Wed, May 03, 2006 at 01:00:28AM +0200, Antonio Cuni wrote: > I'm unsure about using 'delslice': it make me think that such an > operation behave exactly like the corresponding python statement, but > that was not my intent: in my mind the difference between 'remove_range' > and 'delslice' is that the first doesn't handle negative indexes and so > it is likely that the target has some built-in method that implement it > natively. I remember thinking about this that the hint "not negative" could be specified somewhere in addition to the basic oopspec: def ll_getitem_nonneg(func, l, index): ... ll_getitem_nonneg.oopspec = 'list.getitem(l, index)' ll_getitem_nonneg.oopspechint = {'index': '>=0'} # <==== def ll_getitem(func, l, index): ... ll_getitem.oopspec = 'list.getitem(l, index)' The point is that a backend can ignore the hint and implement a general getitem operation that accepts negatives and raises IndexError, or select a more optimized version if it's got one when it sees the hint. Also, note that all our slice operations at RPython level assume that indices are non-negative, with a special exception for exactly x[:-1]. (see rslice.py) A bientot, Armin From arigo at tunes.org Wed May 3 09:27:13 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 3 May 2006 09:27:13 +0200 Subject: [pypy-dev] Seeking students for the Summer of Code In-Reply-To: References: Message-ID: <20060503072712.GA7587@code0.codespeak.net> Hi all, On Wed, May 03, 2006 at 12:15:33AM -0700, Neal Norwitz wrote: > The earlier you submit an application, the more feedback you can get > to improve it and increase your liklihood of getting accepted. I'd like to emphasis this point too :-) The proposals need to somehow give the impression that they are yours, and that your are interested in them and have some idea about what is involved. Feel free to discuss ideas here, and submit proposals early -- you'll get early feedback from us too, and you can update your proposal any number of times until the deadline. Basically, just copy-and-pasting one of the paragraphs from http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas.txt is *not* enough for a proposal! A bientot, Armin. From davidf at sjsoft.com Wed May 3 16:28:18 2006 From: davidf at sjsoft.com (David Fraser) Date: Wed, 03 May 2006 16:28:18 +0200 Subject: [pypy-dev] Signature PEP In-Reply-To: <20060502221146.GB7698@code0.codespeak.net> References: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> <20060502221146.GB7698@code0.codespeak.net> Message-ID: <4458BE02.2010308@sjsoft.com> Armin Rigo wrote: > Hi Seo, > > On Tue, May 02, 2006 at 04:02:46PM +0900, Sanghyeon Seo wrote: > >> What do you think about this Signature PEP? >> http://mail.python.org/pipermail/python-3000/2006-April/001249.html >> > > It's a language design issue so it's off topic :-) > > More seriously, sure, it would be nice to have some kind of standard. > Here, and in many similar code-manipulation places in PyPy, we had to > hack together our own minimal solution. A harder problem along these > lines would be a better way to build pieces of code, in order to create > functions from scratch. \n-joining together source lines full of %s, > avoiding name clashes, and getting the indentation right, is not fun. > > I had a look a while ago at trying to build Python code using ASTs and then compile those, for this very reason. In the end constructing the source was easier, but in theory this is a code-based alternative Cheers David From hpk at trillke.net Wed May 3 16:40:22 2006 From: hpk at trillke.net (holger krekel) Date: Wed, 3 May 2006 16:40:22 +0200 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <5b0248170605021732j383ba149tffdc0b7d1dde9ef1@mail.gmail.com> References: <445713D0.7060906@strakt.com> <20060502095710.45DA67EC9B@postix.sdv.fr> <20060502224019.GC7698@code0.codespeak.net> <5b0248170605021732j383ba149tffdc0b7d1dde9ef1@mail.gmail.com> Message-ID: <20060503144022.GL24602@solar.trillke> Hi Seo, On Wed, May 03, 2006 at 09:32 +0900, Sanghyeon Seo wrote: > 2006/5/3, Armin Rigo : > >Let's consider Javascript (the same argument applies to any interpreted > >language): to write an interpreter, some people focus on the task of > >compiling, say, Javascript source code to get an AST; this is only part > >of the job, and it should be the shortest part -- ideally, finding an > >existing parser would be best, otherwise plugging one together with > >existing tools shouldn't take too long for a regular language like > >Javascript. > > Sadly, parsing Javascript is unnecessarily difficult because of > optional semicolon insertion. Best to look at existing parsers. IMO for a SoC project it would be ok to have a stricter parser. Having javascript programs running on top of PyPy (JsPy or whatever :) being able to run unmodified on other javascript interpreters would be interesting enough. And such an approach would still have the advantages that Armin points out. best, holger From anto.cuni at gmail.com Thu May 4 00:26:12 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 04 May 2006 00:26:12 +0200 Subject: [pypy-dev] Flow graph documentation Message-ID: <44592E04.2010403@gmail.com> Hi all, at last I've begin writing my thesis, though it's still very incomplete and some parts are more similar to a thought flow than to an academic text :-). Since I've not found any doc about flow graphs I've written a bit about them hoping that it could be useful even outside of my thesis. The point is that all I know about flow graphs is deduced from the experience, thus there are possibilities that I've written absurdities. If someone feels like to do quick review of that part my rough draft is here: http://codespeak.net/svn/user/antocuni/tesi/overview.txt thanks ciao Anto From mojobojo at gmail.com Thu May 4 08:29:35 2006 From: mojobojo at gmail.com (Brian Jones) Date: Thu, 4 May 2006 15:29:35 +0900 Subject: [pypy-dev] Seeking students for the Summer of Code In-Reply-To: <20060503072712.GA7587@code0.codespeak.net> References: <20060503072712.GA7587@code0.codespeak.net> Message-ID: <61f323ea0605032329v27be5335o2a34d68b20cca046@mail.gmail.com> On 5/3/06, Armin Rigo wrote: > Hi all, > > On Wed, May 03, 2006 at 12:15:33AM -0700, Neal Norwitz wrote: > > The earlier you submit an application, the more feedback you can get > > to improve it and increase your liklihood of getting accepted. > > I'd like to emphasis this point too :-) The proposals need to somehow > give the impression that they are yours, and that your are interested in > them and have some idea about what is involved. Feel free to discuss > ideas here, and submit proposals early -- you'll get early feedback from > us too, and you can update your proposal any number of times until the > deadline. Basically, just copy-and-pasting one of the paragraphs from > http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas.txt > is *not* enough for a proposal! Hello everyone, I have some questions/problems I'd like to pose in regards to SoC and applying to do a PyPy project. For starters, I am on my second semester of studying abroad in Japan right now, and the semester ends around the beginning-middle of August. I feel that I have plenty of time to devote to a PyPy project, however, many projects I have looked at seem to have a rather strict "This is your #1 priority" policy. This makes sense of course, most of the students applying are probably going into summer break right now. Does this stack against me? Next, most of my programming experience to date is focused around higher level languages (Python, Ruby), and my C knowledge dates back to earlier low level CS courses. I feel comfortable writing something in any language, but would of course be much slower writing C code until I kicked some rust off. I would also want to note that my degree isn't computer related at all. Are either of these another hinderance? Last, is of course a project plan (or lack thereof). I was pointed at PyPy by Chris when I inquired about Stackless several months ago, and have had kind of a backburnered interest in both for awhile now. I would love to contribute to either project, but I really have no idea where my 'niche' would be. Pickling process states so that they can be handed across multiple machines is interesting to me, and even more interesting would be machines with a different target language. I've always liked the idea of platform/language independent objects, although I've never had any idea for practical use (beyond something like wildly dreaming game development). I suppose that even if I cannot participate in SoC, this email is probably a nice start in regards to getting to know the PyPy community. Thanks in advance, Brian Jones From arigo at tunes.org Fri May 5 21:16:21 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri, 5 May 2006 21:16:21 +0200 Subject: [pypy-dev] Seeking students for the Summer of Code In-Reply-To: <61f323ea0605032329v27be5335o2a34d68b20cca046@mail.gmail.com> References: <20060503072712.GA7587@code0.codespeak.net> <61f323ea0605032329v27be5335o2a34d68b20cca046@mail.gmail.com> Message-ID: <20060505191621.GA23019@code0.codespeak.net> Hi Brian, On Thu, May 04, 2006 at 03:29:35PM +0900, Brian Jones wrote: > I have some questions/problems I'd like to pose in regards to SoC and > applying to do a PyPy project. Welcome! > For starters, I am on my second semester of studying abroad in Japan > right now, and the semester ends around the beginning-middle of > August. This is potentially a problem, but maybe not a too serious one if you are usually working on side stuff a lot during semesters. > Next, most of my programming experience to date is focused around > higher level languages (Python, Ruby), and my C knowledge dates back > to earlier low level CS courses. Not a problem for PyPy - we have about 7000 lines of C in a corner but we're working on removing that :-) > Last, is of course a project plan (or lack thereof). I was pointed at > PyPy by Chris when I inquired about Stackless several months ago, and > have had kind of a backburnered interest in both for awhile now. Of course, you need a more concrete project. At first, going through the PyPy documentation should give you some ideas to start with, and then come back to pypy-dev or to the #pypy channel on irc.freenode.net to discuss them. In the area of pickling process or thread states, our current situation is as follows: we are starting to work on it these days (E.U. funding requirements). By the start of the Summer of Code we won't be finished with the basics, though. A bit later -- say by the half of the SoC project duration -- we will hopefully be done with the basics of pickling, and then we will like to move to other topics (we have just too many things to do!). At this point your constribution would be welcome to complete and polish the area (which is a long and never-finished topic, so I don't fear lack of work here :-) So maybe there is a way to submit a SoC proposal that would start with another small PyPy topic to get started, and move to pickling in a second part. At least it would fit. A bientot, Armin. From arigo at tunes.org Fri May 5 21:19:32 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri, 5 May 2006 21:19:32 +0200 Subject: [pypy-dev] Flow graph documentation In-Reply-To: <44592E04.2010403@gmail.com> References: <44592E04.2010403@gmail.com> Message-ID: <20060505191932.GB23019@code0.codespeak.net> Hi Antonio, On Thu, May 04, 2006 at 12:26:12AM +0200, Antonio Cuni wrote: > Since I've not found any doc about flow graphs I've written a bit about > them hoping that it could be useful even outside of my thesis. Ouch! At first sight, your text looks nice, but now we have a merging issue: there is already documentation about flow graphs, it's even a kind-of-well-documented part of PyPy in comparison. It's at http://codespeak.net/pypy/dist/pypy/doc/objspace.html#the-flow-model (I haven't read your docs so far, will come back...) A bientot, Armin. From tismer at stackless.com Sat May 6 11:06:01 2006 From: tismer at stackless.com (Christian Tismer) Date: Sat, 06 May 2006 11:06:01 +0200 Subject: [pypy-dev] Flow graph documentation In-Reply-To: <20060505191932.GB23019@code0.codespeak.net> References: <44592E04.2010403@gmail.com> <20060505191932.GB23019@code0.codespeak.net> Message-ID: <445C66F9.2070301@stackless.com> Armin Rigo wrote: > Hi Antonio, > > On Thu, May 04, 2006 at 12:26:12AM +0200, Antonio Cuni wrote: >> Since I've not found any doc about flow graphs I've written a bit about >> them hoping that it could be useful even outside of my thesis. > > Ouch! At first sight, your text looks nice, but now we have a merging > issue: there is already documentation about flow graphs, it's even a > kind-of-well-documented part of PyPy in comparison. It's at > http://codespeak.net/pypy/dist/pypy/doc/objspace.html#the-flow-model Maybe we need some glossary or other reverse indexes. Certain things are not obvious where to find. -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From elmo13 at jippii.fi Sun May 7 14:02:59 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Sun, 07 May 2006 15:02:59 +0300 Subject: [pypy-dev] An error in doc/getting-started.txt Message-ID: <445DE1F3.1040005@jippii.fi> Here is a diff -u (attached) of a corrected version. The diff is self-explanatory. Elmo -------------- next part -------------- A non-text attachment was scrubbed... Name: getting-started.diff Type: text/x-patch Size: 926 bytes Desc: not available URL: From elmo13 at jippii.fi Sun May 7 20:20:53 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Sun, 07 May 2006 21:20:53 +0300 Subject: [pypy-dev] "compilemodule.py _demo" fails Message-ID: <445E3A85.1070503@jippii.fi> Attached is the output of "py.test ". Apart from that, I'm interested if the few warnings are just irrelevant and will go away eventually, or are they related. Thanks for your great effort, I can't wait till you get the app-level side of this working. Elmo -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test_compilemodule_out.txt URL: From fijal at genesilico.pl Sun May 7 23:57:40 2006 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sun, 07 May 2006 23:57:40 +0200 Subject: [pypy-dev] SoC proposal Message-ID: <445E6D54.8050002@genesilico.pl> I've written down SoC proposal, I has been refactored a little bit since IRC discussion, so I send you all application links. HTML version: http://students.mimuw.edu.pl/~mf197855/application.html Source reStructuredText: http://students.mimuw.edu.pl/~mf197855/application Any comment is very appreciated, also about language. I would like to thank You all with help around that proposal. Regards, Maciek Fijalkowski From janzert at janzert.com Tue May 9 05:25:23 2006 From: janzert at janzert.com (Janzert) Date: Mon, 08 May 2006 23:25:23 -0400 Subject: [pypy-dev] test_all.py results from windows Message-ID: I was asked on #pypy if I would post these here, so here they are. :) http://python.janzert.com/pypy/test_all_26973.txt Sorry if this double posts first time doesn't seem to have taken. Janzert From hpk at trillke.net Tue May 9 12:26:45 2006 From: hpk at trillke.net (holger krekel) Date: Tue, 9 May 2006 12:26:45 +0200 Subject: [pypy-dev] iceland sprint canceled Message-ID: <20060509102645.GQ19139@solar.trillke> Hi folks, as some of you have heard our PyPy Iceland sprint is canceled. There is a chain of events that lead up to this cancelation which is not easy to comprehend or to describe. What it probably boils down to: expectation mismatches and communication problems. A month ago we were asked by a sponsor to freely and quickly select 8 people and work on what we find suitable for PyPy. We planned on this basis and matched it with our 0.9 release goals and explicitely communicated both our goals and participant list to the sprint organizer. We got an OK for it and proceeded with our plannings. Last week it turned out that this OK was not reflecting what the CEO of the sponsoring company thought for the whole sprint event and he was very upset at the situation. This put us into a very unfortunate situation although we had clearly communicated our intentions and had gotten confirmation. We then decided last friday to adapt to the new situation, shifted our sprint goals, planned for explaining the EU some delays and told the sponsor that we were ready to withdraw 4 of the originally intended 8 participants of the sprint. But the CEO somehow perceived our moves as asking "for more" and canceled his funding offer. He had apparently expected us to silently drop the people. Bummer. Of course, we talked a lot yesterday about what went wrong and i (holger) asked myself that even more because of being involved in presenting the modification and "retraction offer" towards the sponsor and probably not hitting the right tone given the situation on the sponsors side. We also think there were misconceptions at play which are based on the difficulties of bringing commercial interests and open source projects together. And certainly some cultural differences. It's interesting to discuss this further, also with respect to possible further endeavours. But maybe not right now. We now are bound to decide soon about an alternative venue for our next PyPy sprint. We'd be happy if the iceland participants and many more can show up. best and a bientot, Holger and Armin From simon at arrowtheory.com Wed May 10 04:32:01 2006 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 10 May 2006 12:32:01 +1000 Subject: [pypy-dev] sprint at europython ? Message-ID: <20060510123201.444d8576.simon@arrowtheory.com> Hi, I'm hoping to come to europython this year; one of the main attractions will be the pypy sprint. Have the dates for this been decided ? cheers, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From mwh at python.net Wed May 10 10:37:02 2006 From: mwh at python.net (Michael Hudson) Date: Wed, 10 May 2006 09:37:02 +0100 Subject: [pypy-dev] Re: sprint at europython ? References: <20060510123201.444d8576.simon@arrowtheory.com> Message-ID: <2mmzdqe0nl.fsf@starship.python.net> Simon Burton writes: > Hi, > > I'm hoping to come to europython this year; one of > the main attractions will be the pypy sprint. Have the > dates for this been decided ? It will be after the conference, probably running for three or four days. > Simon Burton, B.Sc. > Licensed PO Box 8066 > ANU Canberra 2601 > Australia Another aussie at EuroPython? Wow. Cheers, mwh -- okay, tell me if i am crazy you are damn -- from Twisted.Quotes From hpk at trillke.net Thu May 11 08:05:06 2006 From: hpk at trillke.net (holger krekel) Date: Thu, 11 May 2006 08:05:06 +0200 Subject: [pypy-dev] #pypy-sync TODAY 5pm Message-ID: <20060511060506.GX19139@solar.trillke> Hi folks, let's have a developer pypy-sync meeting today (THURSDAY 11TH May) as was mentioned here and there already. Time: 5pm UTC+2 Location: #pypy-sync on freenode Here are the topics, please mail here before if you like to change/amend them: - activity reports (lines of LAST, NEXT and BLOCKERS) - thread cloning approaches As discussed earlier it is urgent to have a working thread cloning approach. Armin and Christian will try to present the approache(s) and time frames for this feature in some written form. - Next PyPy sprint Preliminary discussions resulted in the suggestion to do the next PyPy sprint from 2nd-9th June 2006 and very likely in Duesseldorf (Germany) because that seems to suit most of us best. Topics will be the 0.9 release, ext-compiler, stackless and more or less all topics that have been floating lately. Carl will be trying to fix a place. From cfbolz at gmx.de Thu May 11 15:42:31 2006 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 11 May 2006 15:42:31 +0200 Subject: [pypy-dev] #pypy-sync TODAY 5pm In-Reply-To: <20060511060506.GX19139@solar.trillke> References: <20060511060506.GX19139@solar.trillke> Message-ID: <44633F47.7080506@gmx.de> holger krekel wrote: > let's have a developer pypy-sync meeting today (THURSDAY 11TH > May) as was mentioned here and there already. > > Time: 5pm UTC+2 > Location: #pypy-sync on freenode > > Here are the topics, please mail here before if you > like to change/amend them: sorry, I won't make it, lost too much time with snake :-( my lines: LAST: weakref, tiny stuff, lots of university work NEXT: finally weakref on boehm, starting with __del__ hopefully BLOCKERS: too much at once Cheers, Carl Friedrich From arigo at tunes.org Thu May 11 16:24:13 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 11 May 2006 16:24:13 +0200 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060511060506.GX19139@solar.trillke> References: <20060511060506.GX19139@solar.trillke> Message-ID: <20060511142413.GA2986@code0.codespeak.net> Hi all, On Thu, May 11, 2006 at 08:05:06AM +0200, holger krekel wrote: > - thread cloning approaches > > As discussed earlier it is urgent to have a working > thread cloning approach. Armin and Christian will > try to present the approache(s) and time frames for > this feature in some written form. A word of introduction: we are dividing the work in two directions; one is thread pickling, which is lead by Christian. The other is thread cloning, lead by me. Thread cloning is in principle a subset of what can be done with pickling, as we could pickle a thread, unpickle it, et voila: we have cloned the thread. But pickling is harder and comes with a different set of problems than "just" cloning. It is also likely that pickling threads in random interpreter states will be very difficult, a restriction that does not apply to thread cloning. So we divided the work; the most important reason is that cloning is what the constraint solver developments require urgently. So here is my current point of view on cloning. The goal is to give to RPython code some way to duplicate a "chain of frames", i.e. an RPython coroutine. There are three levels of issues: 1. the interface that RPython code needs to use to do that 2. the selection of what GcStructures must be duplicated or shared 3. the automatic generation of the necessary walkers from the stackless transformer In the following days we should focus on 3. This will need quite some coding efforts by itself, maybe ~ 1 week of dedicated work from where we stand now. Then we need to experiment with various ways to select what needs to be duplicated or not. The problem is that there are some RPython-level and app-level objects that need "obviously" to be shared, like app-level modules, and like interp-level singleton state objects; and others that need obviously to be duplicated so that the newly cloned coroutine has its own copy, like local lists. It's unclear which option we will choose, but Christian has proposed some good ideas. Only experimentation will tell. A possibly good solution that he proposed (but hard to implement efficiently) is to duplicate exactly those objects that have been allocated by the coroutine that we are cloning. We could try to do that *inefficiently*, e.g. by adding an "allocated-by" field to every GcStructs, and see how the result works in practice. Trying that shouldn't be extremely hard; at most 1 more week of dedicated efforts. An advantage of this approach over alternatives is that it's conceptually simpler, and also that the required RPython-level interface is very easy: something along the lines of "newstate = state.fork()". In light of this, it seems that we are at some 2 weeks of hard work from a prototype, in the line of thread cloning. I have intentionally left out the other stackless subject, which is thread pickling; I'll let Christian present it when he's ready to. A bientot, Armin. From elmo13 at jippii.fi Thu May 11 19:17:31 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Thu, 11 May 2006 20:17:31 +0300 Subject: [pypy-dev] Translation segfaults with "compilemodule.py _demo" Message-ID: <446371AB.2060405@jippii.fi> python2.4 pypy/rpython/rctypes/tool/compilemodule.py _demo [cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c ctypesplatcheck_0.c -o ctypesplatcheck_0.o [cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_0.o -lm -lpthread -o /tmp/usession-27/ctypesplatcheck_0 [cbuild:execute] cc -O2 -pthread -c ctypesplatcheck_1.c -o ctypesplatcheck_1.o [cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_1.o -lm -lpthread -o /tmp/usession-27/ctypesplatcheck_1 [translation:info] Annotating&simplifying... [translation:info] with policy: pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy Segmentation fault Other related minor errors: Propably because of the segfault, test.py gives EOFError, which isn't nice =). Something you might know is that using pyhon2.3 I get an "OverflowError: 0 does not fit in signed 64-bit integer" (happens with" from pypy.rpython.lltypesystem import lltype"), which has probably something to do with the resent changes in rarithmetic.py. Should I send the whole tracebacks for the these errors? Elmo From elmo13 at jippii.fi Thu May 11 21:30:37 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Thu, 11 May 2006 22:30:37 +0300 Subject: [pypy-dev] Translation segfaults with "compilemodule.py _demo" In-Reply-To: <446371AB.2060405@jippii.fi> References: <446371AB.2060405@jippii.fi> Message-ID: <446390DD.7090803@jippii.fi> Elmo M?ntynen wrote: >python2.4 pypy/rpython/rctypes/tool/compilemodule.py _demo > >[cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c >ctypesplatcheck_0.c -o ctypesplatcheck_0.o >[cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_0.o -lm >-lpthread -o /tmp/usession-27/ctypesplatcheck_0 >[cbuild:execute] cc -O2 -pthread -c ctypesplatcheck_1.c -o >ctypesplatcheck_1.o >[cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_1.o -lm >-lpthread -o /tmp/usession-27/ctypesplatcheck_1 >[translation:info] Annotating&simplifying... >[translation:info] with policy: >pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy >Segmentation fault > > Segfaults with 2.4 but not with 2.3. The exception in .../pyobj.py:87, which I reported earlier, is still thrown though. >Other related minor errors: >Propably because of the segfault, test.py gives EOFError, which isn't >nice =). > This ---^ is propably py-devel stuff. >Something you might know is that using pyhon2.3 I get an >"OverflowError: 0 does not fit in signed 64-bit integer" (happens with" >from pypy.rpython.lltypesystem import lltype"), which has probably >something to do with the resent changes in rarithmetic.py. > Works now ---^ >Should I send >the whole tracebacks for the these errors? > >Elmo >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev > > From aurelien.campeas at logilab.fr Fri May 12 10:24:42 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Fri, 12 May 2006 10:24:42 +0200 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060511142413.GA2986@code0.codespeak.net> References: <20060511060506.GX19139@solar.trillke> <20060511142413.GA2986@code0.codespeak.net> Message-ID: <20060512082442.GA28391@crater.logilab.fr> On Thu, May 11, 2006 at 04:24:13PM +0200, Armin Rigo wrote: > Hi all, > > On Thu, May 11, 2006 at 08:05:06AM +0200, holger krekel wrote: > > - thread cloning approaches > > > > As discussed earlier it is urgent to have a working > > thread cloning approach. Armin and Christian will > > try to present the approache(s) and time frames for > > this feature in some written form. > > A word of introduction: we are dividing the work in two directions; one > is thread pickling, which is lead by Christian. The other is thread > cloning, lead by me. Thread cloning is in principle a subset of what > can be done with pickling, as we could pickle a thread, unpickle it, et > voila: we have cloned the thread. But pickling is harder and comes with > a different set of problems than "just" cloning. It is also likely that > pickling threads in random interpreter states will be very difficult, a > restriction that does not apply to thread cloning. So we divided the > work; the most important reason is that cloning is what the constraint > solver developments require urgently. Just a note there : the constraint solver doesn't really need thread cloning. It's the framework that makes possible modular integration of constraint solving and logic programming that needs it. Even more off-topic : I've been playing with Mercurial, one in the million new distributed SCM that appears these days (it's Python all over the place of course), and I am pleasantly surprised to discover how the basic concepts of computation spaces seem to match these of an DSCM. Some of the primitives are just the same : clone, merge (with an SCM you have the added ability to arbitrate merge conflicts, with comp. spaces conflicts just mean only one can win), commit (seemingly same semantics, but different usage patterns). This convergence is reassuring, at least on the front of the generality of the aforementioned primitives > > So here is my current point of view on cloning. The goal is to give to > RPython code some way to duplicate a "chain of frames", i.e. an RPython > coroutine. There are three levels of issues: > > 1. the interface that RPython code needs to use to do that > 2. the selection of what GcStructures must be duplicated or shared > 3. the automatic generation of the necessary walkers from the > stackless transformer > > In the following days we should focus on 3. This will need quite some > coding efforts by itself, maybe ~ 1 week of dedicated work from where we > stand now. > > Then we need to experiment with various ways to select what needs to be > duplicated or not. The problem is that there are some RPython-level and > app-level objects that need "obviously" to be shared, like app-level > modules, and like interp-level singleton state objects; and others that > need obviously to be duplicated so that the newly cloned coroutine has > its own copy, like local lists. Applevel modules are the responsibility of their implementors : if they contain shared global mutable state, then sharing them makes them thread-unsafe and unsuitable for use in comp. spaces. But that's life. well, I cannot but ask : would it be possible to be able to clone these anyway ? Now I wonder if built-in modules will need a make-them-thread-safe pass (this question is motivated by these interp-level singleton state objects you mention). > > It's unclear which option we will choose, but Christian has proposed > some good ideas. Only experimentation will tell. A possibly good > solution that he proposed (but hard to implement efficiently) is to > duplicate exactly those objects that have been allocated by the > coroutine that we are cloning. We could try to do that *inefficiently*, > e.g. by adding an "allocated-by" field to every GcStructs, and see how > the result works in practice. Trying that shouldn't be extremely hard; > at most 1 more week of dedicated efforts. An advantage of this approach > over alternatives is that it's conceptually simpler, and also that the > required RPython-level interface is very easy: something along the lines > of "newstate = state.fork()". > > In light of this, it seems that we are at some 2 weeks of hard work from > a prototype, in the line of thread cloning. Happy to hear this :) Thanks for all of this. Aur?lien. From arigo at tunes.org Fri May 12 11:05:37 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri, 12 May 2006 11:05:37 +0200 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060512082442.GA28391@crater.logilab.fr> References: <20060511060506.GX19139@solar.trillke> <20060511142413.GA2986@code0.codespeak.net> <20060512082442.GA28391@crater.logilab.fr> Message-ID: <20060512090537.GA26683@code0.codespeak.net> Hi Aurelien, On Fri, May 12, 2006 at 10:24:42AM +0200, Aur?lien Camp?as wrote: > Just a note there : the constraint solver doesn't really need thread > cloning. It's the framework that makes possible modular integration of > constraint solving and logic programming that needs it. I was about to ask precisely if the operation I described here makes sense from your point of view. That's basically what you need, isn't it? > Applevel modules are the responsibility of their implementors : if > they contain shared global mutable state, then sharing them makes them > thread-unsafe and unsuitable for use in comp. spaces. But that's > life. > > well, I cannot but ask : would it be possible to be able to clone > these anyway ? It's part of the experimentation that we'll need. They would be cloned if we go for the approach that all objects *created* by a thread are cloned together with the thread. This includes app-level objects. This interpretation follows from a nice -- if vague -- high-level description from Christian: if you start a thread and it computes something up to some point, and clone it as this point, then you get a new thread that "looks like" it has been started from scratch and then ran up to the same point, repeating the same computations. Of course, this is vague because we need to explain that input-output and other side effects that the thread might have had are not duplicated. A bientot, Armin. From aurelien.campeas at logilab.fr Fri May 12 12:42:17 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Fri, 12 May 2006 12:42:17 +0200 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060512090537.GA26683@code0.codespeak.net> References: <20060511060506.GX19139@solar.trillke> <20060511142413.GA2986@code0.codespeak.net> <20060512082442.GA28391@crater.logilab.fr> <20060512090537.GA26683@code0.codespeak.net> Message-ID: <20060512104217.GB28391@crater.logilab.fr> On Fri, May 12, 2006 at 11:05:37AM +0200, Armin Rigo wrote: Hi Armin, > Hi Aurelien, > > On Fri, May 12, 2006 at 10:24:42AM +0200, Aur?lien Camp?as wrote: > > Just a note there : the constraint solver doesn't really need thread > > cloning. It's the framework that makes possible modular integration of > > constraint solving and logic programming that needs it. > > I was about to ask precisely if the operation I described here makes > sense from your point of view. That's basically what you need, isn't > it? Roughly I think so. Some funny details may be discovered later of course. Well, clone != fork in my mind. We really want a *copy*. Clone or copy are good words for this. Absolutely avoid to share stuff (unless we know for sure sharing is safe but that would be an implementation detail). (fork is confusingly related to some unixish syscalls whose precise semantics are no more clear to me at this points -- it's been a long time since I've directly used it). Especially, cloning a space is different from 'newspace', which is much more akin to the fork operator. I don't have the time right now to expand on this & will try to make things more clear next week. I'm still unsure about the semantics of newspace... > > > Applevel modules are the responsibility of their implementors : if > > they contain shared global mutable state, then sharing them makes them > > thread-unsafe and unsuitable for use in comp. spaces. But that's > > life. > > > > well, I cannot but ask : would it be possible to be able to clone > > these anyway ? > > It's part of the experimentation that we'll need. They would be cloned > if we go for the approach that all objects *created* by a thread are > cloned together with the thread. This includes app-level objects. I'm not sure I understand what another approach would look like... We could restrict the programming style for code to be running inside comp. spaces but if it is possible to effectively clone everything (for some ill-defined notion of everything) then it'l be fine. I'm not 100% sure. Here are some of the constraints : In principle, side-effects on the parent space and the outer world should be forbidden from within a running comp. space ('cause the jury is still out on its outcome). The problem is not that we want to clone app-level modules (most Python code tend to be decomposed into many modules, I don't believe we want to forbid that) but that we will, by doing so, allow the computation running inside a space to do unwanted stuff, like sneakily doing I/O for instance, or indirectly calling some fancy C code that does God-knows-what. Something like a "restricted execution" environment would be interesting to have. In Mozart/Oz, for what it's worth, they 'just' filter any attempt to mutate the upper/outside world. I suspect it comes with a price, at least in terms of implementation complexity (I've not actually seen the code though). They do this also because comp. spaces in Oz are more, hmmm, expressive that what we'll have in PyPy : the 'newspace' is not scheduled to go into PyPy (well, at this point). Its usefulness is unclear to us. So, yes, we want to clone app-level modules, and generally, that everything created by a thread be cloned (first). We'll see later what can be made to prevent insanity to happen from within spaces. > > This interpretation follows from a nice -- if vague -- high-level > description from Christian: if you start a thread and it computes > something up to some point, and clone it as this point, then you get a > new thread that "looks like" it has been started from scratch and then > ran up to the same point, repeating the same computations. Of course, > this is vague because we need to explain that input-output and other > side effects that the thread might have had are not duplicated. The Oz way is to leave IO to the 'top-level' space. What really counts is the internal state of the threads. Whether we would like a 'logic/relational program' to perform IO is a topic for another day (I suspect currently we expect such a program to act on local state, where local means 'belongs 100% to the Python world' (but what does/doesn't exactly ? I don't know for sure). Oh, one last point : in my view, cloning one thread means cloning a whole tree of threads. Confusedly & hastily yours Hope this helps a bit. Aur?lien. From picxenk at gmail.com Sun May 14 10:42:56 2006 From: picxenk at gmail.com (SeungBum Kim) Date: Sun, 14 May 2006 17:42:56 +0900 Subject: [pypy-dev] New logo for pypy Message-ID: Hi, all I created a prototype for new pypy logo before. Some of you like a recursive aspect from my design. Old one was.. http://image.xenbio.net/logo/pypy_proto I made another one. It also has a recursive concept, but a little bit abstract image. http://image.xenbio.net/logo/neopypy1 Have fun. From mwh at python.net Mon May 15 11:38:49 2006 From: mwh at python.net (Michael Hudson) Date: Mon, 15 May 2006 10:38:49 +0100 Subject: [pypy-dev] Duesseldorf PyPy sprint 2-9 June 2006 Message-ID: <2mzmhj8w5y.fsf@starship.python.net> The next PyPy sprint will be held in the Computer Science department of Heinrich-Heine Universitaet Duesseldorf from the 2nd to the 9th of June. The main focus of the sprint will be on the goals of the upcoming June 0.9 release: * The extension module compiler. The goal is to be able to use a single RPython source code as an extension module for both PyPy and CPython. The means to get there is -- most likely -- by compiling ctypes-based modules into either pypy-c or a CPython dll/so. * Write some more modules in this style with ctypes. * Stackless: the big missing feature is pickling running tasklets. There is also some smaller work that needs to be done, like exposing all the existing RPython-level interfaces to app-level (e.g. greenlets). * Write more documentation. * Misc topics, depending on interests: back-ends (CLI, Squeak), testing framework, modified semantics (security/sandboxing...), etc. If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2006/people.html Cheers, mwh -- Slim Shady is fed up with your shit, and he's going to kill you. -- Eminem, "Public Service Announcement 2000" From mwh at python.net Thu May 18 01:11:09 2006 From: mwh at python.net (Michael Hudson) Date: Thu, 18 May 2006 00:11:09 +0100 Subject: [pypy-dev] pypy-sync tomorrow! Message-ID: <2mfyj86ycy.fsf@starship.python.net> It's Thursday and that means it's pypy-sync time. I'd like to invite all active pypy developers to the #pypy-sync channel on freenode at 5:30 PM GMT+2 tomorrow (well, today really by now, 2006-05-18). The known-at-this-point topics are: 1) activity reports as usual 2) oopsla paper We should submit a paper to the Dynamic Languages Symposium 2006 which is attached to this year's OOPSLA: http://www.dcl.hpi.uni-potsdam.de/dls2006/openconf.php What should be in it, who should write it, etc. The deadline is June 1. Some background reading would be overview papers on self, jikes, ... 3) state of the world There has been quite a lot of development over the past few weeks, and it's been fairly hard to keep up with it all. It would be good to have a quick chat about where things are, esp the things targetted for the 0.9 release. If you have any other topics, please email them here beforehand. Cheers, mwh -- People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work if it just results in what I consider to be a better system. -- Linus Torvalds From tismer at stackless.com Thu May 18 16:16:49 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu, 18 May 2006 16:16:49 +0200 Subject: [pypy-dev] pypy-sync: in case that I'm not there at 5pm Message-ID: <446C81D1.8090306@stackless.com> Hi friends, we are invited to an open-air concert, today. If it does not start raining, again, then I will miss the pypy-sync meeting. DONE: coached Stephan on app-level Stackless, helped Eric with the pickling support, analyzed many flow-graphs NEXT: support for costate for app-level, more coaching, implement a few demo picklers, investigate more on feasibility of the pickling design doc, prepare for Iceland sprint BLOCK: I cannot estimate what will come out of it. I'm not sure if I can find a way to avoid pickling of many records. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From anto.cuni at gmail.com Sun May 21 12:37:42 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 21 May 2006 12:37:42 +0200 Subject: [pypy-dev] gencli checkin In-Reply-To: <446E2E0C.2040707@genesilico.pl> References: <446E2E0C.2040707@genesilico.pl> Message-ID: <447042F6.90708@gmail.com> Hi Maciek, hi all Maciek Fijalkowski wrote: > I've just checked in my changes to gencli which are essential for genjs > to use gencli. It does not brake any more tests (7 failed as was > before). I hope I don't get in conflict with your actual edits. sorry for the very late response; I'm cc:ing to pypy-dev so that others can read. I've read your changes, good work: I think that now the best thing to do is to collect the modules used by both backends in a proper place, e.g. a genoo package or so (better names are welcome :-). Here are some comments on the modules that might be moved to genoo: class_.py, function.py, record.py: they contain both generic code and gencli specific code (such as Class.get_base_class() and and Class._ctor() methods). We could put generic 'Class', 'Function' and 'Record' class in genoo and subclass it in the backends. cts.py: this is CLI specific, but each backend need a similar module: we should at least put the interface specification in genoo, and possibly some shared code. database.py: there are both generic and specific parts, too. We should provide a more flexible way to handle constants, and maybe we can integrate it with the code from gensqueak that implements a sort of priority queue for emitting things in the right order. IMHO each backends should be able to decide whether to emit things in sequential order (as gencli does, should be faster) or in "declaration order" (as gensqueak and genjs2 need). ilgenerator.py: same as cts.py. oopspec.py: same as class_.py, function.py, record.py: some code can be shared, some need to be backend-specific. rte.py: its goal is to automatically compile a DLL every time the source if modified: this could be useful for all backends that need to compile some runtime environment "offline". sdk.py: very cli-specific. metavm.py: the code here is of three types: - generic code that can be used by all backends: the Generator/MicroInstrucion/InstructionList architecture is designed to be as much generic as possible, so it's not a problem to reuse it (note that the Generator interface is not very up-to-date, see Function.py). - some classes are not cli-specific but they have been designed for emitting code on stack-based machines; I guess they are not very useful for genjs2, but they could be useful for a hypotetic genjava, for example. - some code is very cli-specific, such as the _RuntimeNew class. I wrote some ideas on how you can use and extend metavm for your goal, but I decided to move them to a fresh e-mail so we don't mix the two discussions. At last we need a way to tell genoo where to find the code specific to each backend. I've seen that at the moment you decided to separately pass things such as CTS, Function, opcodes etc as arguments to GenCli constructor, but perhaps it would be better to collect them in a sort of 'container' class and pass this all around; something like this: class CliFunction: ... class CliClass: ... class CTS: ... cli_opcodes = { ... } class CliBackend: Function = CliFunction Class = CliClass TypeSystem = CTS opcodes = cli_opcodes ... Better names other than CliBackend are welcome, too :-). What do you think about this design? Valentino, Nikh, Seo and other that are working on high level backends: do you think that such a genoo could be useful for gensqueak and gencl, too? ciao Anto From anto.cuni at gmail.com Sun May 21 12:38:40 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 21 May 2006 12:38:40 +0200 Subject: [pypy-dev] MetaVM design Message-ID: <44704330.1020609@gmail.com> Hi Maciek, I all as I announced in my previous e-mail here are some ideas on how to use metavm for genjs2. I think the best way to use metavm for your purposes is to write a bunch of MicroInstruction classes designed for emitting source code. For example I guess that some rpython low level instruction need to be mapped to infix operator (such as int_add & co.) while others need to be mapped to some support function (such as int_abs). You could write something like this: class InfixOperator(MicroInstruction): def __init__(self, operator): self.operator = operator def render(self, generator, op): generator.emit('(%s %s %s)' %\ (op.args[0], self.operator, op.args[1]) class CallHelper(MicroInstruction): def __init__(self, helper): self.helper = helper def render(self, generator, op): arglist = ... # compute arglist from op.args generator.emit('%s(%s)' % (self.helper, arglist) Then, in your opcodes.py: opcodes = { 'int_add': InfixOperator('+'), 'int_abs': CallHelper('abs') } The difficult thing here is to design the Generator interface in a way that is suitable for emitting both asm code and source code. Maybe we try to separate the two worlds, in this way: genoo/metavm.py --------------- class Generator: def function_signature(self, graph): pass # put here all methods shared by both AsmGenerator and SourceGenerator class StackBasedGenerator(Generator): def emit(self, instr, *args): pass def call(self, func_name): pass def load(self, v): pass def store(self, v): pass ... class SourceCodeGenerator(Generator): def emit(self, expression): pass # and so on genoo/function.py ----------------- class Function(Node, Generator): # some shared code ... class StackBasedFunction(Node, StackBasedGenerator): ... class SourceCodeFunction(Node, SourceCodeGenerator): ... gencli/function.py ------------------ class CliFunction(StackBasedFucntion): ... genjs2/function.py ------------------ class JsFunction(SourceCodeFunction): ... Once we have this design, each MicroInstruction subclass can decide whether to target the generic Generator interface or one of its subinterfaces. ciao Anto From simon at arrowtheory.com Wed May 24 06:02:07 2006 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 24 May 2006 14:02:07 +1000 Subject: [pypy-dev] llvm backend Message-ID: <20060524140207.4bc81933.simon@arrowtheory.com> Hi, I'm exploring the llvm backend of pypy. Does it work without llvm-gcc ? I tried the example at doc/getting-started.html and it seems to first compile to c code then invoke llvm-gcc. >>> f = t.compile_llvm() (calls to llvm-gcc) But I'm also looking at the source for the llvm translator and there is code there for generating llva directly. Now I'm trying to run the demos in pypy/translator/llvm/demo. I assume I can use my regular python to run these.. somehow.. $ python run.py Traceback (most recent call last): ... File "/mnt/hda9/simonb/local/pypy-svn/pypy/annotation/bookkeeper.py", line 16, in ? from pypy.annotation.listdef import ListDef, MOST_GENERAL_LISTDEF ImportError: cannot import name ListDef ? Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From simon at arrowtheory.com Wed May 24 06:40:21 2006 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 24 May 2006 14:40:21 +1000 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <20060524144021.086eea94.simon@arrowtheory.com> On Wed, 24 May 2006 14:02:07 +1000 Simon Burton wrote: > > But I'm also looking at the source for the llvm translator and there is > code there for generating llva directly. I'm gradually finding more pieces of the puzzle: 1) there is a wrapper to LLVM's ExecutionEngine.ParseAssemblyString 2) there are calls to ee.parse in pyllvm/test/test_ee.py, but only with test snippets but then: 3) there is code for generating llva in codewriter.py 4) calls to codewriter come from *node.py modules, these stitch together the pypy flow and type objects into llva code 5) genllvm.py coordinates all of this and then calls to buildllvm It seems that steps 1 and 2 are yet to be used as buildllvm uses the commandline llvm-as at the moment. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From dooms at info.ucl.ac.be Wed May 24 09:06:20 2006 From: dooms at info.ucl.ac.be (=?ISO-8859-1?Q?Gr=E9goire_Dooms?=) Date: Wed, 24 May 2006 09:06:20 +0200 Subject: [pypy-dev] pypy-dev Digest, Vol 310, Issue 5 In-Reply-To: References: Message-ID: <447405EC.7000007@info.ucl.ac.be> > Date: Fri, 12 May 2006 12:42:17 +0200 > From: Aur?lien Camp?as > Subject: Re: [pypy-dev] Thread cloning (coroutine cloning, really) > To: Armin Rigo > Cc: pypy-dev at codespeak.net > Message-ID: <20060512104217.GB28391 at crater.logilab.fr> > Content-Type: text/plain; charset=iso-8859-1 > > We could restrict the programming style for code to be running inside > comp. spaces but if it is possible to effectively clone everything > (for some ill-defined notion of everything) then it'l be fine. I'm not > 100% sure. Here are some of the constraints : > > In principle, side-effects on the parent space and the outer world > should be forbidden from within a running comp. space ('cause the jury > is still out on its outcome). > If I can throw in my two cents, you should really try to remove this limitation of Mozart. This is a real pain in the ass when you want to know what happens in a space (I used to need that in Mozart for debugging or statistics). In Mozart it is possible to use a hack in order to have side effects out of a space (ie modifiy state outside the space) by using ports. I think a design which would allow it would be much more Pythonic (we are between consenting adults :-) Best, -- Gr?goire From aurelien.campeas at logilab.fr Wed May 24 09:54:36 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Wed, 24 May 2006 09:54:36 +0200 Subject: [pypy-dev] pypy-dev Digest, Vol 310, Issue 5 In-Reply-To: <447405EC.7000007@info.ucl.ac.be> References: <447405EC.7000007@info.ucl.ac.be> Message-ID: <20060524075436.GB7772@crater.logilab.fr> On Wed, May 24, 2006 at 09:06:20AM +0200, Gr?goire Dooms wrote: > > >Date: Fri, 12 May 2006 12:42:17 +0200 > >From: Aur?lien Camp?as > >Subject: Re: [pypy-dev] Thread cloning (coroutine cloning, really) > >To: Armin Rigo > >Cc: pypy-dev at codespeak.net > >Message-ID: <20060512104217.GB28391 at crater.logilab.fr> > >Content-Type: text/plain; charset=iso-8859-1 > > > > >We could restrict the programming style for code to be running inside > >comp. spaces but if it is possible to effectively clone everything > >(for some ill-defined notion of everything) then it'l be fine. I'm not > >100% sure. Here are some of the constraints : > > > >In principle, side-effects on the parent space and the outer world > >should be forbidden from within a running comp. space ('cause the jury > >is still out on its outcome). > > > > If I can throw in my two cents, you should really try to remove this > limitation of Mozart. This is a real pain in the ass when you want to > know what happens in a space (I used to need that in Mozart for > debugging or statistics). In Mozart it is possible to use a hack in > order to have side effects out of a space (ie modifiy state outside the > space) by using ports. > I think a design which would allow it would be much more Pythonic (we > are between consenting adults :-) I had been thinking about that indeed. Glad you tell us what you think. Anyway, by default/as a starting point PyPy's comp spaces will be "open" ; as seen in Mozart with ports or whatever it must be hard to restrict operations enough to ensure a proper sealing of a space and in the case of Python, worse than hard ... > > Best, > -- > Gr?goire > From eric at vanrietpaap.nl Wed May 24 10:02:30 2006 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Wed, 24 May 2006 10:02:30 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <68A555EA-B3F4-4971-BFD5-2C9569B71957@vanrietpaap.nl> Hi Simon, llvm-gcc is used to compile the external function C code (that genllvm shares with genc) to a .ll file. The code in llvm/demo does not seem to work at the moment, sorry for that :( It looks like a circular dependency issue. You assumption about running the files in ..../demo with regular python is correct, however the run.py is respronsible for the invokation of genc/genllvm in this case so you would have to do something like: "python bpnn.py". About your other mail about genllvm: We indeed explored the usage of llvm's jit feature with ExecutionEngine.ParseAssemblyString but the use of this is currently limited to pypy/jit/codegen/llvm , where genllvm generates .ll code that gets added to the llvm jit with the execution engine. The steps the genllvm follows are: - (when necessary) create a .ll file out of genc's external function c code (one day to be replaced by (r)ctypes code) - generate and attach our source file - run llvm-as to convert the .ll to a .bc file - run opt to optimize with llvm's transformation - generate (a) assembly with llvm's native (x86) backend or (b) c code again with llvm's c backend - in case of b we run this code through gcc to further (profile based) optimize and create the final executable. I can image this information is a bit to dense so if you have any more questions please ask them here. It's good for us to know what people require for getting a better overview. cheers Eric On May 24, 2006, at 6:02 AM, Simon Burton wrote: > > Hi, > > I'm exploring the llvm backend of pypy. Does it work without llvm- > gcc ? > > I tried the example at doc/getting-started.html and it seems to first > compile to c code then invoke llvm-gcc. > >>>> f = t.compile_llvm() > (calls to llvm-gcc) > > But I'm also looking at the source for the llvm translator and > there is > code there for generating llva directly. > > Now I'm trying to run the demos in pypy/translator/llvm/demo. > I assume I can use my regular python to run these.. somehow.. > > $ python run.py > Traceback (most recent call last): > ... > File "/mnt/hda9/simonb/local/pypy-svn/pypy/annotation/ > bookkeeper.py", line 16, in ? > from pypy.annotation.listdef import ListDef, MOST_GENERAL_LISTDEF > ImportError: cannot import name ListDef > > ? > > Simon. > > > -- > Simon Burton, B.Sc. > Licensed PO Box 8066 > ANU Canberra 2601 > Australia > Ph. 61 02 6249 6940 > http://arrowtheory.com > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From arigo at tunes.org Wed May 24 10:36:32 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 10:36:32 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <20060524083632.GA27954@code0.codespeak.net> Hi Simon, On Wed, May 24, 2006 at 02:02:07PM +1000, Simon Burton wrote: > $ python run.py > Traceback (most recent call last): > ... > File "/mnt/hda9/simonb/local/pypy-svn/pypy/annotation/bookkeeper.py", line 16, in ? > from pypy.annotation.listdef import ListDef, MOST_GENERAL_LISTDEF > ImportError: cannot import name ListDef Oups, mea culpa. I added an import at the top of run.py without checking that it was still working. Importing pypy.annotation.listdef before importing the other modules of that package creates a circular import bug. As a workaround to this particular cycle, I added an import line in the __init__ of the package. A bientot, Armin From eric at vanrietpaap.nl Wed May 24 10:43:13 2006 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Wed, 24 May 2006 10:43:13 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <3DEDC750-E75A-440E-BC9E-50122FF003D1@vanrietpaap.nl> Simon, hi! Something I have to add to my last email... The profile based optimization stuff is only the the nightly benchmark cronjob (pypy/translator/goal/bench-cronjob.py) but could in principle (with proper support from genc and genllvm) be added as an option to ./translate.py . cheers Eric From arigo at tunes.org Wed May 24 10:50:04 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 10:50:04 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524083632.GA27954@code0.codespeak.net> References: <20060524140207.4bc81933.simon@arrowtheory.com> <20060524083632.GA27954@code0.codespeak.net> Message-ID: <20060524085004.GB27954@code0.codespeak.net> ... On Wed, May 24, 2006 at 10:36:32AM +0200, Armin Rigo wrote: > As a workaround to this particular cycle, I added an import line in the > __init__ of the package. Now that I think about it, there are a few places (including this cycle) where modules import other modules only for the side-effects of having something registered; e.g. pypy.annotation.model imports pypy.annotation.unaryop and pypy.annotation.binaryop, and pypy.rpython.rtyper imports many of the pypy.rpython.r* modules. There is the same problem with every place that uses the extregistry (currently only rctypes and some ootypesystem functions and types). Maybe we could reorganize that by moving the necessary import statements to the __init__ of the appropriate packages. It certainly looks clearer and easier to control than these import loops. (It would also require us to finally do the long-pending clean-up of dividing the 'rpython' package in two: one for the rtyper and one for the modules of stuff that RPython programs are allowed to call, e.g. rarithmetic and objectmodel.) A bientot, Armin From arigo at tunes.org Wed May 24 13:47:28 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 13:47:28 +0200 Subject: [pypy-dev] Security ideas Message-ID: <20060524114728.GA6374@code0.codespeak.net> Hi all, On Monday I was at an inspiring seminar about (a specific form of) language-level security. I've collected the PyPy-ification of these ideas there: http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt Although the focus is different, it makes me think that we could also use similar ideas to implement a form of 'rexec' (restricted execution), with functions compiled by secure() as in the draft above, but running at a priviledge level which is lower than the default ambiant level instead of higher. A bientot, Armin From jacob at strakt.com Wed May 24 14:21:54 2006 From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Wed, 24 May 2006 14:21:54 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <20060524114728.GA6374@code0.codespeak.net> References: <20060524114728.GA6374@code0.codespeak.net> Message-ID: <200605241421.55683.jacob@strakt.com> On Wednesday 24 May 2006 13:47, Armin Rigo wrote: > Hi all, > > On Monday I was at an inspiring seminar about (a specific form of) > language-level security. I've collected the PyPy-ification of these > ideas there: > > http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt > > Although the focus is different, it makes me think that we could also > use similar ideas to implement a form of 'rexec' (restricted execution), > with functions compiled by secure() as in the draft above, but running > at a priviledge level which is lower than the default ambiant level > instead of higher. This is quite interesting, but I have some concerns over the scheme presented. It seems to only take into consideration who gets to see the contents of an object. However, real information security is just as often concerned with who gets to set or modify the contents of an object. This produces security classifications that can't be represented as a linear scale, leading to a much more complex infrastructure for determining what classification to give to an object that receives it from multiple parents. Jacob From arigo at tunes.org Wed May 24 19:12:07 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 19:12:07 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <200605241421.55683.jacob@strakt.com> References: <20060524114728.GA6374@code0.codespeak.net> <200605241421.55683.jacob@strakt.com> Message-ID: <20060524171206.GA22544@code0.codespeak.net> Hi Jacob, On Wed, May 24, 2006 at 02:21:54PM +0200, Jacob Hall?n wrote: > This is quite interesting, but I have some concerns over the scheme presented. > It seems to only take into consideration who gets to see the contents of an > object. However, real information security is just as often concerned with > who gets to set or modify the contents of an object. This produces security > classifications that can't be represented as a linear scale, leading to a > much more complex infrastructure for determining what classification to give > to an object that receives it from multiple parents. Definitely. The scheme presented doesn't propose any classification, and it allows us to experiment with variants. As presented, the levels don't have to form a linear order but only have to be arranged in a semi-lattice (i.e. where "public" is the most permissive, and for any two levels L1 and L2 we can form a higher level "L1 union L2" that represents someone with both L1 and L2 credential levels). Moreover nothing is specified about what the level means and where exactly they are enforced, so we can definitely have object attributes that require different levels to be read or modified. There are many theories available; I've heard of one where each level is actually a set of tuples, where each tuple gives an "owner" name and a set of names of persons that can read the information. Also, it's easy to come up with anything custom in this approach -- as opposed to other approaches where changing the theory requires a whole compiler to be rewritten, syntax to be redesigned, etc. A bientot, Armin. From mwh at python.net Wed May 24 23:21:08 2006 From: mwh at python.net (Michael Hudson) Date: Wed, 24 May 2006 22:21:08 +0100 Subject: [pypy-dev] [pypy-svn] r27660 - in pypy/branch/njriley-trans/pypy: rpython/lltypesystem translator/c In-Reply-To: <20060524184814.8B3941006E@code0.codespeak.net> (njriley@codespeak.net's message of "Wed, 24 May 2006 20:48:14 +0200 (CEST)") References: <20060524184814.8B3941006E@code0.codespeak.net> Message-ID: <2mac973yrf.fsf@starship.python.net> njriley at codespeak.net writes: > Author: njriley > Date: Wed May 24 20:48:13 2006 > New Revision: 27660 > > Modified: > pypy/branch/njriley-trans/pypy/rpython/lltypesystem/rclass.py > pypy/branch/njriley-trans/pypy/translator/c/exceptiontransform.py > pypy/branch/njriley-trans/pypy/translator/c/node.py > Log: > Make exceptions thread-local Would this change make sense on the trunk too? Cheers, mwh -- Hmmm... its Sunday afternoon: I could do my work, or I could do a Fourier analysis of my computer's fan noise. -- Amit Muthu, ucam.chat (from Owen Dunn's summary of the year) From njriley at uiuc.edu Wed May 24 23:39:38 2006 From: njriley at uiuc.edu (Nicholas Riley) Date: Wed, 24 May 2006 16:39:38 -0500 Subject: [pypy-dev] [pypy-svn] r27660 - in pypy/branch/njriley-trans/pypy: rpython/lltypesystem translator/c In-Reply-To: <2mac973yrf.fsf@starship.python.net> References: <20060524184814.8B3941006E@code0.codespeak.net> <2mac973yrf.fsf@starship.python.net> Message-ID: <20060524213938.GA7462@uiuc.edu> On Wed, May 24, 2006 at 10:21:08PM +0100, Michael Hudson wrote: > njriley at codespeak.net writes: > > > Author: njriley > > Date: Wed May 24 20:48:13 2006 > > New Revision: 27660 > > > > Modified: > > pypy/branch/njriley-trans/pypy/rpython/lltypesystem/rclass.py > > pypy/branch/njriley-trans/pypy/translator/c/exceptiontransform.py > > pypy/branch/njriley-trans/pypy/translator/c/node.py > > Log: > > Make exceptions thread-local > > Would this change make sense on the trunk too? Assuming the GIL model, I don't see anywhere that the exception state gets swapped on a thread switch now; so, if this is the case, then yes, they'd make sense. With actual concurrent execution as I'm doing, that wouldn't help, of course. My changes would be fine if we could expect __thread (or some equivalent compiler-managed TLS) to be available everywhere, but PyPy doesn't make that assumption (see Samuele's work, e.g. pypy.translator.tool.cbuild.check_under_under_thread). The work I did for the old, pre-exceptiontransform model used pthread thread-specific variables, which would be possible as an alternative, but it'd take some work to automatically generate, and it'd impose considerably higher overhead on exception accesses. -- Nicholas Riley | From hpk at trillke.net Thu May 25 16:10:26 2006 From: hpk at trillke.net (holger krekel) Date: Thu, 25 May 2006 16:10:26 +0200 Subject: [pypy-dev] pypy-sync Monday 29th 5pm UTC+2 Message-ID: <20060525141026.GL12245@solar.trillke> Hi folks, the DDorf sprint, EuroPython talk deadlines and the 0.9 release are all jointly and quickly approaching. Let's have a brief #pypy-sync meeting Monday afternoon 5pm UTC+2, 29th May with these topics and try to co-ordinate last preps and efforts. And please have your "last/next/blockers" lines ready. have a nice weekend! best, holger From faassen at infrae.com Fri May 26 16:26:19 2006 From: faassen at infrae.com (Martijn Faassen) Date: Fri, 26 May 2006 16:26:19 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <20060524114728.GA6374@code0.codespeak.net> References: <20060524114728.GA6374@code0.codespeak.net> Message-ID: <4477100B.8050602@infrae.com> Armin Rigo wrote: > Hi all, > > On Monday I was at an inspiring seminar about (a specific form of) > language-level security. I've collected the PyPy-ification of these > ideas there: > > http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt > > Although the focus is different, it makes me think that we could also > use similar ideas to implement a form of 'rexec' (restricted execution), > with functions compiled by secure() as in the draft above, but running > at a priviledge level which is lower than the default ambiant level > instead of higher. As a general note it might be useful to talk to Jim Fulton for real-world experience concerning language-level security in Python. I'll cc him so he at least is aware of your security ideas document. In Zope 2, there is a precompiler for untrusted Python code, offering, as far as I understand, true language-level security. In Zope 3 this approach has been dropped as hard to maintain and replaced with object level security (attribute access is controlled with a permission system). Regards, Martijn From pedronis at strakt.com Sat May 27 17:59:22 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Sat, 27 May 2006 17:59:22 +0200 Subject: [pypy-dev] pypy-sync Monday 29th 5pm UTC+2 In-Reply-To: <20060525141026.GL12245@solar.trillke> References: <20060525141026.GL12245@solar.trillke> Message-ID: <4478775A.4090409@strakt.com> holger krekel wrote: > Hi folks, > > the DDorf sprint, EuroPython talk deadlines and > the 0.9 release are all jointly and quickly approaching. > > Let's have a brief > > #pypy-sync meeting Monday afternoon 5pm UTC+2, 29th May > > with these topics and try to co-ordinate last preps > and efforts. And please have your "last/next/blockers" > lines ready. > I'm going to be traveling back to Gothenburg at that time on monday. Before leaving for the vacation I helped on various gc/stackless stuff and experimented with an idea that may be useufl for tasklet unpickling. Samuele. From nicolas.chauvat at logilab.fr Mon May 29 17:45:16 2006 From: nicolas.chauvat at logilab.fr (Nicolas Chauvat) Date: Mon, 29 May 2006 17:45:16 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <20060524114728.GA6374@code0.codespeak.net> References: <20060524114728.GA6374@code0.codespeak.net> Message-ID: <20060529154516.GE28271@crater.logilab.fr> On Wed, May 24, 2006 at 01:47:28PM +0200, Armin Rigo wrote: > On Monday I was at an inspiring seminar about (a specific form of) > language-level security. I've collected the PyPy-ification of these > ideas there: > > http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt Let me add a reference, just in case it could be useful to future readers of the archives: http://en.wikipedia.org/wiki/E_programming_language -- Nicolas Chauvat logilab.fr - services en informatique avanc?e et gestion de connaissances From fijal at genesilico.pl Mon May 29 19:44:47 2006 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 29 May 2006 19:44:47 +0200 Subject: [pypy-dev] Ootypesystem based genjs (genjs2) Message-ID: <447B330F.4080804@genesilico.pl> I've added what I've done in last week with genjs, altough I'm not very happy with some issues. - LoopFinder does not find nested whiles (I think it's also gensquek problem) - Inheritance is not implemented - Dict support is not implemented - There is some code redundancy, because of copying from gencli. I hope to refactor it with antonio at DDorf sprint. - I don't like builtin method and function mapping solution, especially copying whole misc.js at the beginning of source file. Some function should be copied on higher, some on lower level. From fijal at genesilico.pl Mon May 29 20:04:24 2006 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 29 May 2006 20:04:24 +0200 Subject: [pypy-dev] SoC Message-ID: <447B37A8.8020908@genesilico.pl> Maybe that's a little bit late, but I've got broadband finally. I would like to thank all of pypy team for helping me with my SoC. From anto.cuni at gmail.com Mon May 29 20:22:40 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 29 May 2006 20:22:40 +0200 Subject: [pypy-dev] SoC In-Reply-To: <447B37A8.8020908@genesilico.pl> References: <447B37A8.8020908@genesilico.pl> Message-ID: <447B3BF0.6050502@gmail.com> Maciek Fijalkowski wrote: > Maybe that's a little bit late, but I've got broadband finally. > > I would like to thank all of pypy team for helping me with my SoC. A big 'thank you all' from me, too. I hope we'll be able to repay your trust by completing our works nicely :-). ciao Anto From l.oluyede at gmail.com Mon May 29 21:13:02 2006 From: l.oluyede at gmail.com (Lawrence Oluyede) Date: Mon, 29 May 2006 21:13:02 +0200 Subject: [pypy-dev] SoC In-Reply-To: <447B3BF0.6050502@gmail.com> References: <447B37A8.8020908@genesilico.pl> <447B3BF0.6050502@gmail.com> Message-ID: <9eebf5740605291213q9933a49rc9f9e935fd36b8a6@mail.gmail.com> On 5/29/06, Antonio Cuni wrote: > Maciek Fijalkowski wrote: > > Maybe that's a little bit late, but I've got broadband finally. > > > > I would like to thank all of pypy team for helping me with my SoC. > > A big 'thank you all' from me, too. I hope we'll be able to repay your > trust by completing our works nicely :-). I'm really lucky being part of the SoC project and have the opportunity to do something useful for the community of the language I use and love :-) Thanks guys. -- Lawrence http://www.oluyede.org/blog From anto.cuni at gmail.com Tue May 30 11:06:42 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 30 May 2006 11:06:42 +0200 Subject: [pypy-dev] Missed pypy-sync Message-ID: <447C0B22.2090200@gmail.com> Hi all, I've realized just now that I've missed the pypy-sync yesterday... sorry! Btw, here is my activity report: LAST: work on unifying rpython/ and cli/ tests; basic string support in gencli NEXT: DDorf sprint BLOCKERS: uni stuffs ciao Anto