From anto.cuni at gmail.com Tue Jul 3 15:56:51 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 03 Jul 2007 15:56:51 +0200 Subject: [pypy-dev] Alternative algorithm for attribute lookup Message-ID: <468A55A3.4080009@gmail.com> Hi all, I passed the last few days slowly experimenting with alternative/faster ways to do lookup of attributes. For simplicity, I didn't touch the interpreter but I simply wrote few rpython programs doing only the last part of the attribute lookup, i.e. accessing the underlying RPython dictionary. Moreover, I played only with dict of strings, since this is how objects are implemented when multidict is enabled. My first experiment was with precomputed hashes for keys; I introduced a new method get_fast on dictionaries, which accepts both the key and its hash value, so it does not need to compute it. Of course this is just a quick hack because programs using get_fast works only when translated and can't run on top of CPython, but it was the simplest way to experiment. You can find both the get_fast patch and the benchmark here: http://codespeak.net/svn/user/antocuni/lookup-hack/get_fast.patch http://codespeak.net/svn/user/antocuni/lookup-hack/targethashcache.py The interesting thing is the result of the benchmark: since RPython strings already cache their hash I didn't expect get_fast to be much faster than get, but the benchmark says the opposite; on my machine: antocuni at anto lookup-hack $ ./targethashcache-c 50000000 iterations get: 3.820000 seconds get_fast: 2.520000 seconds get_fast: 1.515873 times faster It is 50% faster. I really don't know why, maybe there is something wrong in the benchmark or maybe something is wrong with the hash cache of rpython strings. The second and more insteresting experiment is a bit more involved. The idea is that for most of Python objects, we can guess a set of "likely attributes" at compile time: - for modules, by inspecting its global namespace; - for classes, by inspecting the namespace of a class definition; - for instances, by looking for LOAD_FAST 0, STORE_ATTR x inside the methods of the class. Note that we don't pretend to catch all the possible attributes, just the most likely. The assumption is that it's very uncommon to have a "non-likely attribute" and that for most of the objects the set of the attributes at runtime is a subset of the likely attributes. The idea is to speed up consistently all the accesses to likely attributes, at the cost of a slighly slow down for accessing the non-likely ones. Once we have collected the set of likely attributes, we compute a perfect hash function for this set; for my experiment, I used an hacked version of this algorithm: http://www.amk.ca/files/python/perfect_hash.txt The perfect hash function looks like this, where N and G vary from set to set of likely attributes:: def perfect_hash(N, G, key): h = hash(key) # normal hash i = h & N-1 j = i+1 res = G[i] + G[j] res = res & N-1 return res Moreover, since h depends only on key we could also precomputed it at compile time, as we did for get_fast:: def perfect_hash(N, G, key, h): ... Finally, we insert a fast path for the common case in the lookup code:: def lookup(d, key, h): # h is the precomputed hash index = perfect_hash(d.N, d.G, key, h) entry = d.entries[index] if entry.is_valid() and entry.key is key: return entry.value else: # do a full lookup Note that in the fast path we use 'is' instead of == to compare the keys; since the W_Strings storing the names of the attributes are interned, this should be enough. Note also that G and N are stored on the dict (thus we can have different perfect hashes for different set of likely attributes, as we need). If my assumption about likely attributes is true, most of LOAD_ATTR will follow the fast path, resulting in a considerable speedup: if the object only uses likely attributes, there will never be a collision. Most important, this does not break anything, because in case the fast path is not followed (for example when we do {get,set}attr(obj, 'attr')), the full lookup will work as well. You can find the benchmark here (this doesn't need get_fast.patch): http://codespeak.net/svn/user/antocuni/lookup-hack/targetperfectdict.py For the test, I implemented a perfect_dict as an RPython class, thus it might be slower than what we can get if we implement it as a real rpython type on the style of r_dict; in particular, RPython does index-checking when calcultaing G[i] and G[j], but it's not necessary. Although perfect_dict is not as fast as possible, the result of the benchmark is very interesting: antocuni at anto lookup-hack $ ./targetperfectdict-c 50000000 iterations get: 3.740000 seconds pdict: 1.450000 seconds pdict: 2.579310 times faster What do you think about this idea? Is there any obvious bug/wrong assumption I can't catch? Do you think it might be worth of implementing it? (it would be a nice sprint topic). ciao Anto From scott+pypy-dev at scottdial.com Wed Jul 4 22:02:21 2007 From: scott+pypy-dev at scottdial.com (Scott Dial) Date: Wed, 04 Jul 2007 16:02:21 -0400 Subject: [pypy-dev] select/test_readable on Win32 blocks forever Message-ID: <468BFCCD.6040201@scottdial.com> Greetings, I haven't figured this one out. As the subject says, test_readable for the select module gets stuck forever. If I trust the history of the bot, then it appeared somewhere after r44639 and before r44646. I've looked through the revisions and nothing jumped out at me.. happy hunting to someone more aware of the changes. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From wchunming at gmail.com Thu Jul 5 07:47:25 2007 From: wchunming at gmail.com (=?GB2312?B?zfW0usP3?=) Date: Thu, 5 Jul 2007 13:47:25 +0800 Subject: [pypy-dev] debug(read source code and test) pypy in pydev Message-ID: <64e4dfb0707042247n7818ecdfvd051a827abc29e3@mail.gmail.com> Hi, I am a newbie to python and pypy. I choose to read pypy source code to learn about python and python library. But when I try to run pypy in debug mode in pydev, I got the following problem: /// error message begin pydev debugger Traceback (most recent call last): File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd.py", line 754, in debugger.run(setup['file'], None, None) File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd.py", line 597, in run execfile(file, globals, locals) #execute the script File "E:\Java\pySamples\pypy\bin\py.py", line 14, in from pypy.tool import option File "E:\Java\pySamples\pypy\tool\option.py", line 4, in from pypy.config.pypyoption import get_pypy_config File "E:\Java\pySamples\pypy\config\pypyoption.py", line 2, in import py, os File "E:\Java\pySamples\pypy\bin\py.py", line 14, in from pypy.tool import option ImportError: cannot import name option Exception in thread pydevd.Writer (most likely raised during interpreter shutdown): Traceback (most recent call last): File "C:\Python25\lib\threading.py", line 460, in __bootstrap File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd_comm.py", line 258, in run /// error message end I try to figure out why this exception or error come out. I found that py.py import option.py, and option.py import pypyoption.py, and pypyoption.py import py.py again. Is this the problem of pypy(I don't think so), or the problem of pydev debugger, or something else? why? Any comments or assistance that can light up the way are much appreciated.Thanks. Wang Chunming -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Thu Jul 5 14:41:16 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 5 Jul 2007 14:41:16 +0200 Subject: [pypy-dev] debug(read source code and test) pypy in pydev In-Reply-To: References: <64e4dfb0707042247n7818ecdfvd051a827abc29e3@mail.gmail.com> Message-ID: Hello, Wang Chunming wrote: > Hi, > I am a newbie to python and pypy. I choose to read pypy source code > to learn about python and python library. But when I try to run pypy > in debug mode in pydev, I got the following problem: [Traceback] > > I try to figure out why this exception or error come out. I found > that py.py import option.py, and option.py import pypyoption.py, and > pypyoption.py import py.py again. Is this the problem of pypy(I don't > think so), or the problem of pydev debugger, or something else? why? > Any comments or assistance that can light up the way are much > appreciated.Thanks. There are two "py" involved here: - you are running the py.py script - the other is the py library: http://codespeak.net/py/dist/ Normally, this does not cause any problem, but Pydev seems to tweak the pythonpath and adds several entries... Try to set the current directory to E:\Java\pySamples\pypy\bin\ (In eclipse, open the "Run/Debug..." window, and select the "Arguments tab") Or you could just rename py.py to something else (runpy.py for example). Note that because of a python bug, you cannot use python 2.5 to debug pypy. (A strange interaction between generator destructors, the sys.settrace function and the current thread state. www.python.org/sf/1733973 shows a simple case, filed by the pydev developer) Python 2.4 seems to work correctly. Hope this helps, -- Amaury Forgeot d'Arc From amauryfa at gmail.com Thu Jul 5 15:28:21 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 5 Jul 2007 15:28:21 +0200 Subject: [pypy-dev] select/test_readable on Win32 blocks forever In-Reply-To: <468BFCCD.6040201@scottdial.com> References: <468BFCCD.6040201@scottdial.com> Message-ID: Hello Scott, Scott Dial wrote: > Greetings, > > I haven't figured this one out. As the subject says, test_readable for > the select module gets stuck forever. If I trust the history of the bot, > then it appeared somewhere after r44639 and before r44646. I've looked > through the revisions and nothing jumped out at me.. happy hunting to > someone more aware of the changes. When I wrote these tests a few months ago, they passed on the two windows machines where I ran them (Win2k + python2.5 behind a firewall, and XP + python2.4 at home). They still pass without problem (except for some network operations forbidden by the firewall). Is it possible that the bot runs in a restricted environment? What is the configuration of this machine? I suggest to run the following test function, either alone, or somewhere in the test suite Could you put it near the top of pypy/module/select/test/test_select.py ? It has nothing to do with pypy, but repeats the same test at the python level. # Sanity check of the standard Python interpreter def test_CPython_select(): import thread, socket, select sock = socket.socket() try_ports = [1023] + range(20000, 30000, 437) for port in try_ports: print 'binding to port %d:' % (port,), sockaddress = ('127.0.0.1', port) try: sock.bind(sockaddress) print 'works' break except socket.error, e: print e else: raise e else: assert False, "Could not create server" sock.listen(1) writeend = socket.socket() thread.start_new_thread(writeend.connect, (sockaddress,)) readend, addr2 = sock.accept() try: iwtd, owtd, ewtd = select.select([readend], [], [], 0) assert iwtd == owtd == ewtd == [] writeend.send('X') iwtd, owtd, ewtd = select.select([readend], [], []) assert iwtd == [readend] assert owtd == ewtd == [] print "Test OK" finally: writeend.close() readend.close() -- Amaury Forgeot d'Arc From jgustak at gmail.com Fri Jul 6 08:55:50 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 6 Jul 2007 08:55:50 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Hello. What we have done on scheme interpreter: Working lambdas, with closures almost as described in r5rs. What we miss here is proper tail-recursion and issues, when lambda is called with wrong number of arguments (though should be simple to add). Some rethinking on ExecutionContext. Now we have global and local scope (looks like python). When copying ExecutionContext only local scope is copied, global one is shared. What more, copied ExecutionContext cannot bound new variables in global scope, though it can set new values to existing ones. Done quotations both as (quote ) and as '. After all it looks like distinction between W_Identifier and W_Symbol is useless. So probably only one of them will live in the future. Also implemented parsing for dotted lists (needed for lambdas definition). cons, car, cdr were easy ones. Probably not much hacking before the sprint. So the plan for EP sprint is RPython. I would like also to discuss some other features, if time will let us. See you soon on #pypy or on EP. Cheers, Jakub From cfbolz at gmx.de Mon Jul 9 19:33:30 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 9 Jul 2007 19:33:30 +0200 Subject: [pypy-dev] paper on runtime specialization Message-ID: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> Hi all! Today I found this rather interesting paper on runtime specialization of C: http://www.cs.washington.edu/research/dyncomp/Papers/tr-97-03-03-abstract.html I have only skimmed it so far, but it seems there are many shared ideas between PyPy and this approach. Cheers, Carl Friedrich From cfbolz at gmx.de Mon Jul 9 20:58:33 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 9 Jul 2007 20:58:33 +0200 Subject: [pypy-dev] paper on runtime specialization In-Reply-To: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> References: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> Message-ID: <348899050707091158x1f36d4fdxacc990f1e9d46ca7@mail.gmail.com> 2007/7/9, Carl Friedrich Bolz : > Today I found this rather interesting paper on runtime specialization of C: There is even more interesting stuff (including papers) on their web-page: http://www.cs.washington.edu/research/dyncomp/ Cheers, Carl Friedrich From fijal at genesilico.pl Wed Jul 11 13:13:40 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 11 Jul 2007 13:13:40 +0200 Subject: [pypy-dev] Quick personal view regarding js interpreter Message-ID: <4694BB64.5050700@genesilico.pl> It's completely unusable. Usually you end up with Fatal RPythonError in case you try to run stuff. This is not how we go about stuff! All error reporting should work well not only on top of js_interactive, but actually on top of js-c as well. Leonardo - have you even run js_interactive once before you commit? Last time it was unbound local or so. Cheers, fijal :. From scott+pypy-dev at scottdial.com Wed Jul 11 16:56:20 2007 From: scott+pypy-dev at scottdial.com (Scott Dial) Date: Wed, 11 Jul 2007 10:56:20 -0400 Subject: [pypy-dev] Win32 tester Message-ID: <4694EF94.2010908@scottdial.com> Greetings Everyone, I am going to have to stop running the win32 test suite. I am not sure there is anyone actively correcting problems on win32, so it may be of no consequence. If someone else is interested in running this, then I would be more than glad to share my knowledge with them. Otherwise, sorry to leave you without. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From bea at changemaker.nu Thu Jul 12 15:29:20 2007 From: bea at changemaker.nu (=?ISO-8859-1?Q?Beatrice_D=FCring?=) Date: Thu, 12 Jul 2007 15:29:20 +0200 Subject: [pypy-dev] Win32 tester In-Reply-To: <4694EF94.2010908@scottdial.com> References: <4694EF94.2010908@scottdial.com> Message-ID: <46962CB0.8030409@changemaker.nu> Hi there Scott Dial skrev: > Greetings Everyone, > > I am going to have to stop running the win32 test suite. I am not sure > there is anyone actively correcting problems on win32, so it may be of > no consequence. If someone else is interested in running this, then I > would be more than glad to share my knowledge with them. Otherwise, > sorry to leave you without. > > -Scott > > Thanks for helping out with this Scott! And the door is always open - as you know ;-) Cheers Bea From fijal at genesilico.pl Fri Jul 13 13:14:00 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 13 Jul 2007 13:14:00 +0200 Subject: [pypy-dev] Win32 tester In-Reply-To: <4694EF94.2010908@scottdial.com> References: <4694EF94.2010908@scottdial.com> Message-ID: <46975E78.7030007@genesilico.pl> Scott Dial wrote: > Greetings Everyone, > > I am going to have to stop running the win32 test suite. I am not sure > there is anyone actively correcting problems on win32, so it may be of > no consequence. If someone else is interested in running this, then I > would be more than glad to share my knowledge with them. Otherwise, > sorry to leave you without. > > -Scott > > I would blame a bit (from my side) lack of a single web page showing all failures on all platforms. Unless we have one, it's probably a bit useless to have such a status page, just because. I would like to set up one at some point, but that's another issue. Thanks for cooperating and I (personally) used it from time to time (although never said a word). Cheers, fijal :. From gbowyer at fastmail.co.uk Fri Jul 13 15:37:57 2007 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Fri, 13 Jul 2007 14:37:57 +0100 Subject: [pypy-dev] Taking pypy for a test drive Message-ID: Sorry if this is an exceptional newbie post ... Now that pypy has hit v1 I have tried using it for a few test examples, most of this stuff so far is a little bit simple, however one test is to run a django application that I am developing for someone with pypy rather that cpython However when I fire up pypy i get the following from the django application, what am I missing here greg at gregslaptop ~/projects/minesite/minesite $ ./pypy-c manage.py runserver debug: WARNING: library path not found, using compiled-in sys.path Traceback (most recent call last): File "?", line 32, in run_toplevel File "manage.py", line 11, in execute_manager(settings) File "/usr/lib/python2.4/site-packages/django/core/management.py", line 1736, in execute_manager execute_from_command_line(action_mapping, argv) File "/usr/lib/python2.4/site-packages/django/core/management.py", line 1626, in execute_from_command_line translation.activate('en-us') File "/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", line 66, in activate return real_activate(language) File "/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", line 29, in delayed_loader from django.conf import settings File "/usr/lib/python2.4/site-packages/django/conf/__init__.py", line 149, in __builtins__['_'] = first_time_gettext TypeError: cannot set items on object From exarkun at divmod.com Fri Jul 13 15:50:59 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 13 Jul 2007 09:50:59 -0400 Subject: [pypy-dev] Taking pypy for a test drive In-Reply-To: Message-ID: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> On Fri, 13 Jul 2007 14:37:57 +0100, Greg Bowyer wrote: >Sorry if this is an exceptional newbie post ... > >Now that pypy has hit v1 I have tried using it for a few test examples, >most of this stuff so far is a little bit simple, however one test is to >run a django application that I am developing for someone with pypy >rather that cpython > >However when I fire up pypy i get the following from the django >application, what am I missing here > >greg at gregslaptop ~/projects/minesite/minesite $ ./pypy-c manage.py runserver >debug: WARNING: library path not found, using compiled-in sys.path >Traceback (most recent call last): > File "?", line 32, in run_toplevel > File "manage.py", line 11, in > execute_manager(settings) > File "/usr/lib/python2.4/site-packages/django/core/management.py", >line 1736, in execute_manager > execute_from_command_line(action_mapping, argv) > File "/usr/lib/python2.4/site-packages/django/core/management.py", >line 1626, in execute_from_command_line > translation.activate('en-us') > File >"/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", >line 66, in activate > return real_activate(language) > File >"/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", >line 29, in delayed_loader > from django.conf import settings > File "/usr/lib/python2.4/site-packages/django/conf/__init__.py", line >149, in > __builtins__['_'] = first_time_gettext >TypeError: cannot set items on object It seems likely that what django ought to be doing is: import __builtin__ __builtin__._ = first_time_gettext And a similar change in first_time_gettext. __builtin__/__builtins__ is a bit subtle and I might be forgetting something, but I think the above is correct. I remember that __builtins__ is somethings a dict instead of the __builtin__ module, but I forget when (and I couldn't find the case with a few simple tests). I think python-dev has pointed out that this is an implementation detail of CPython on several occassions, though, and that explicitly importing the __builtin__ module is the right thing to do. Jean-Paul From njriley at uiuc.edu Fri Jul 13 16:07:10 2007 From: njriley at uiuc.edu (Nicholas Riley) Date: Fri, 13 Jul 2007 09:07:10 -0500 Subject: [pypy-dev] Taking pypy for a test drive In-Reply-To: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> References: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> Message-ID: <20070713140710.GA14844@uiuc.edu> On Fri, Jul 13, 2007 at 09:50:59AM -0400, Jean-Paul Calderone wrote: > It seems likely that what django ought to be doing is: > > import __builtin__ > __builtin__._ = first_time_gettext > > And a similar change in first_time_gettext. __builtin__/__builtins__ is > a bit subtle and I might be forgetting something, but I think the above > is correct. I remember that __builtins__ is somethings a dict instead of > the __builtin__ module, but I forget when (and I couldn't find the case > with a few simple tests). I think python-dev has pointed out that this > is an implementation detail of CPython on several occassions, though, and > that explicitly importing the __builtin__ module is the right thing to > do. It's documented in the Python reference manual. -- Nicholas Riley | From simon at arrowtheory.com Tue Jul 17 03:22:03 2007 From: simon at arrowtheory.com (Simon Burton) Date: Mon, 16 Jul 2007 18:22:03 -0700 Subject: [pypy-dev] revision 45089 Message-ID: <20070716182203.0d1953e1.simon@arrowtheory.com> I found that revision 45089 makes compilation much slower. No idea why. Simon. ------------------------------------------------------------------------ r45089 | justas | 2007-07-14 08:35:59 -0700 (Sat, 14 Jul 2007) | 8 lines (Arlo, Justas) Introduce --pyversion option to select the parser/compiler grammar Defaults to 2.4, also accepts 2.3 and 2.5a. Changed a bunch of imports to use the selected grammar. TODO: pypy/interpreter/pyparser/symbol.py uses a fixed list of symbols, and it shouldn't From arigo at tunes.org Tue Jul 17 11:57:09 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 17 Jul 2007 11:57:09 +0200 Subject: [pypy-dev] revision 45089 In-Reply-To: <20070716182203.0d1953e1.simon@arrowtheory.com> References: <20070716182203.0d1953e1.simon@arrowtheory.com> Message-ID: <20070717095709.GA11010@code0.codespeak.net> Hi Simon, On Mon, Jul 16, 2007 at 06:22:03PM -0700, Simon Burton wrote: > I found that revision 45089 makes compilation much slower. No idea why. Thanks for the note. I fixed it in r45153 after a cProfile run of some tests - the problem was obscure: the flow object space was taking ages (about 0.2 seconds for simple functions). This is because the parser and compiler instantiation code, which is not needed at all for the flow space, was actually called many times per function. And r45089 made that code take much longer. It's not a problem in general because it's not supposed to happen frequently - indeed, it does not need to be called at all when we use the flow space. A bientot, Armin. From cfbolz at gmx.de Tue Jul 17 14:26:08 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 17 Jul 2007 14:26:08 +0200 Subject: [pypy-dev] sprint report Message-ID: <469CB560.9030108@gmx.de> Hi all! Since I didn't manage to come to Europython and the sprint afterwards, I would really appreciate it if somebody wrote a sprint report. Since I guess that is kind of unlikely to happen now, could at least everybody write a paragraph about what he worked on? Cheers, Carl Friedrich From micahel at gmail.com Tue Jul 17 15:20:36 2007 From: micahel at gmail.com (Michael Hudson) Date: Tue, 17 Jul 2007 14:20:36 +0100 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: On 17/07/07, Carl Friedrich Bolz wrote: > Hi all! > > Since I didn't manage to come to Europython and the sprint afterwards, I > would really appreciate it if somebody wrote a sprint report. Since I > guess that is kind of unlikely to happen now, could at least everybody > write a paragraph about what he worked on? Samuele and I hacked on the twisted mess of memory management implementations. Basically we managed to much simplify the interface the low-level backends need to provide -- there is no longer any need to support any 'malloc' operation that deals with types. There is still confusion between various malloc flavours and operations, but it's getting better. Backends that support Boehm need to implement three new operations: boehm_malloc, boehm_malloc_atomic and boehm_register_finalizer. These could/should one day be removed and implemented with rffi (they're just function calls to named function^Wmacros, after all), but that seemed like too much wizardry for the time we had. Cheers, mwh From simon at arrowtheory.com Tue Jul 17 17:02:53 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 17 Jul 2007 08:02:53 -0700 Subject: [pypy-dev] expected_extra_mallocs Message-ID: <20070717080253.a1387d0e.simon@arrowtheory.com> Here is a little test: def func(): x = 1. print x def test_1(): from pypy.translator.c.test.test_genc import compile _func = compile(func, []) _func() I get this error: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def checking_fn(*args, **kwds): if 'expected_extra_mallocs' in kwds: expected_extra_mallocs = kwds.pop('expected_extra_mallocs') else: expected_extra_mallocs = 0 res = compiled_fn(*args, **kwds) mallocs, frees = module.malloc_counters() E assert mallocs - frees == expected_extra_mallocs > assert (6 - 5) == 0 [/home/simon/local/pypy-latest/pypy/translator/c/test/test_genc.py:54] So, do I call this with expected_extra_mallocs=1 ? In general, how many extra mallocs should I expect ? Simon. From jacob at openend.se Tue Jul 17 19:33:32 2007 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 17 Jul 2007 19:33:32 +0200 Subject: [pypy-dev] More sprint reporting Message-ID: <200707171933.32223.jacob@openend.se> Here is what I know happened in other groups: Armin and Laura worked on rffi. After 2 days of trying different approaches and getting stumped each time, they made a breakthrough on Saturday, and managed to produce a fairly complete slution in a matter of a few hours. Making the solution more complete would require a rewrite of ctypes (Ick!). Maciek worked on getting PyPy to be self hosting under pypy-c. This succeeded before he left the sprint. Jakub worked on more scheme features. With the help of Antonio, he managed to get quite a bit done. Exactly what he did, I don't know. Christian worked on fixing pickling of tasklets. He got as far as understanding what the real problem was before I left. Michael and Samuele worked on malloc cleanup. I think they were fairly satisfied with the result. Samuele found and fixed a very nasty bug somewhere in the translator toolchain. Exception transformation was dependent on the order the graphs were evaluated. The result was that exceptions could be lost without a trace. There were 3 people who's names I have forgotten who worked on better documentation on how to leverage the javascript backend. There were some results checked in, but I don't know how complete it is. I think Holger did some work on improving py.test. That was at least his plan, but I never got a chance to ask what he actually did. I think that is about it. Jacob From jacob at openend.se Tue Jul 17 19:38:39 2007 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 17 Jul 2007 19:38:39 +0200 Subject: [pypy-dev] Sprint report Message-ID: <200707171938.39921.jacob@openend.se> I think this got stuck in moderation, as I sent it with a from address that didn't match my mailinglist subscription. Justas, Arlo Belshee and I worked on Python 2.5 compliance. We started with the "with" statement and fairly quickly found that it seemed to work as intended, except for the fact that the necessary "from __future__ import with_statement" had no effect. To make this work properly, we decided to make a clean implementation - CPython does nasty things in order to avoid modifying the AST on the fly. This required modifying the grammar, which in turn led to the discovery that building a parser from the grammar lacked unit tests. We then created the neccessary test setup and made tests for the modifications we were making for the grammar. Then we went on to being able to insert a rule for the "with" statement into the builder on the fly. This required more instrumentation for unit testing and some modifications to the builder. Finally, added a config option to allow selecting different versions from the command line. This was essentially finished, though there may still be places which have the Pthon version hard coded. Jacob From jgustak at gmail.com Tue Jul 17 21:44:24 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Tue, 17 Jul 2007 21:44:24 +0200 Subject: [pypy-dev] More sprint reporting In-Reply-To: <200707171933.32223.jacob@openend.se> References: <200707171933.32223.jacob@openend.se> Message-ID: Here is what happened with scheme interpreter: Antonio and I were working on scheme interpreter to translate. - (antocuni, jlg) The biggest issue was with number handling, it resulted with numbers hierarchy rethinking and arithmetical operations refactoring most of them are dynamically generated. - (fijal, jlg) Some effort was put into syntax checking, especially in macros where lot of w_pair.car/cdr had to be checked if are still instance of W_Pair. - (antocuni, jlg) Then there was work on delayed evaluation. During this Armin found bug in lambda definition identical to one I made with W_Promise. - (arigo, jlg) Finally we made proper tail-recursion work for calls in tail context. - (arigo, jlg) Thats it! btw. I made some dummy benchmark on tail-recursive loop and some arithmetical operations. Our ss-c is about 5 times slower than guile. Cheers, Jakub From simon at arrowtheory.com Tue Jul 17 22:42:38 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 17 Jul 2007 13:42:38 -0700 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <20070717134238.a1c0d9ab.simon@arrowtheory.com> I worked with Maciek on generating rffi wrappers from ctypes objects. The first method we tried used code generation. Later on I got it to generate live rffi objects from the ctypes objects, without code generation. I also made some progress on generating rffi wrappers for cairo and SDL, and also testing these on ll2ctypes. Simon. From fijal at genesilico.pl Wed Jul 18 11:54:49 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 18 Jul 2007 11:54:49 +0200 Subject: [pypy-dev] Sprint report In-Reply-To: <200707171938.39921.jacob@openend.se> References: <200707171938.39921.jacob@openend.se> Message-ID: <469DE369.7010905@genesilico.pl> Three guys who's names Jacob doesn't remember were working on flex backend (reusing JavaScript one to be able to produce flex code and some libraries). I've got no idea what was their progress (hopefully they're reading pypy-dev and are able to reply :-) Cheers, fijal :. From lac at openend.se Wed Jul 18 12:05:03 2007 From: lac at openend.se (Laura Creighton) Date: Wed, 18 Jul 2007 12:05:03 +0200 Subject: [pypy-dev] Sprint report In-Reply-To: Message from Maciek Fijalkowski of "Wed, 18 Jul 2007 11:54:49 +0200." <469DE369.7010905@genesilico.pl> References: <200707171938.39921.jacob@openend.se> <469DE369.7010905@genesilico.pl> Message-ID: <200707181005.l6IA53ih016094@theraft.openend.se> In a message of Wed, 18 Jul 2007 11:54:49 +0200, Maciek Fijalkowski writes: >Three guys who's names Jacob doesn't remember were working on flex >backend (reusing JavaScript one to be able to produce flex code and some >libraries). I've got no idea what was their progress (hopefully they're >reading pypy-dev and are able to reply :-) > >Cheers, >fijal I think they were Ren? Dudfield, Alejandro Curia and Lucio Torre, from Australia, Argentina and Argentina, who may not be home yet. Laura From anto.cuni at gmail.com Wed Jul 18 18:13:20 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 18 Jul 2007 18:13:20 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <469E3C20.7010803@gmail.com> Carl Friedrich Bolz wrote: > Hi all! > > Since I didn't manage to come to Europython and the sprint afterwards, I > would really appreciate it if somebody wrote a sprint report. Since I > guess that is kind of unlikely to happen now, could at least everybody > write a paragraph about what he worked on? On the first day I worked with Jakub to make the scheme interpreter translatable. On the second day, I paired with Maciek trying to make pypy-c self-hosted: we fixed few bugs and now it is self-hosted, as long as pypy-c is being translated with the same opcodes as the hosting pypy-c. Finally, I spent the third day by working again with Jakub on the scheme interpreter and by experimenting with method lookup in the interpreter, without concluding anything interesting :-). ciao Anto From simon at arrowtheory.com Fri Jul 20 03:31:17 2007 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 19 Jul 2007 18:31:17 -0700 Subject: [pypy-dev] rffi strict typing Message-ID: <20070719183117.97ad1ce3.simon@arrowtheory.com> The lltype's are very strict about types. If I declare an llexternal that takes a Float arg, I cannot call it with an int arg. Well maybe that's not so bad. But in rlib.rsdl there are functions that take Unsigned, and these don't work even if called with a non-negative constant int... Can we loosen up these restrictions ? Maybe put an "is_compatable(self, other)" method on LowLevelType's ? Simon. using py lib: /home/simon/local/pypy-latest/py test/test_SDL.py[2] FF __________________________________________________________________________________________________________________________________________________________________________________________________________ entrypoint: test_run ____________________________________________________________ def test_run(): > demo() [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:48] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def demo(): > if SDL.Init(SDL.INIT_VIDEO) < 0: [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:13] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __call__(self, *args): if isinstance(self._T, FuncType): if len(args) != len(self._T.ARGS): raise TypeError,"calling %r with wrong argument number: %r" % (self._T, args) for a, ARG in zip(args, self._T.ARGS): if typeOf(a) != ARG: E raise TypeError,"calling %r with wrong argument types: %r" % (self._T, args) > TypeError: calling Signed> with wrong argument types: (32,) ... From jgustak at gmail.com Fri Jul 20 11:39:28 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 20 Jul 2007 11:39:28 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: What's on with scheme interpreter: Nice progress during post EuroPython sprint, see: http://codespeak.net/pipermail/pypy-dev/2007q3/003919.html And after the sprint: (define (fun ) ) syntactic sugar added to lambda definitions. (begin ...), (letrec ...) added. This leads to small changes in ExecutionContext (set-car! ), (set-cdr! ) added. Careful, it is possible to create circural list, and crash interpreter when it will try to display such list. Added quasi-quotations. Code is not so straightforward. Maybe it can be done differently. W_Identifier and W_Symbol are now W_Symbol. All symbols are created by symbol(name) function, which stores all symbols in global W_Symbol.obarray dictionary or returns existing one. It makes comparing symbols trivial. Next step is to implement macros. There was also plan to move some some of the syntax checking to parser: e.g. (if test then else) to get parsed directly to MacroIf(test, then, else). The problem is, that this syntax can change in scheme on runtime: e.g. (define if #t) so it doesn't make much sense. And later adding main primitive procedures will make scheme interpreter quite usable. Cheers, Jakub From arigo at tunes.org Fri Jul 20 14:48:17 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:48:17 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <20070720124817.GA19336@code0.codespeak.net> Hi all, As Laura and Jacob already hinted at, Laura and me managed to write three times and trash twice code that allows ll external functions to be really called on top of CPython. This means that code can now be written using rffi, including calls to external functions, and this can be tested on top of CPython. We have thus a ctypes replacement, from the point of the view of the CPython user; and we no longer need rctypes at all from the point of the view of the RPython user. Our code (in ll2ctypes.py) is able to transparently convert structures created by lltype.malloc(TYPE, flavor='raw') so that their storage becomes a ctypes object internally. This is done lazily, when the structure "escapes" to a call to an external function. The call itself is also performed through ctypes. As usual we had to fight a bit with ctypes to have it do what we really need (instead of what it thinks you'd prefer today) but nothing deep (Jacob's mail seems to say that we were blocked unless we rewrote ctypes - I'm not sure what he refers to). This is a no-API piece of code invoked automatically when an lltype pointer to an external function is called on top of CPython. A few new pieces of API have been added to rffi.py for general rffi usage, notably rffi.cast(TARGETTYPE, value) which tries to have the same semantics as the C-level cast operator. A bientot, Armin. From arigo at tunes.org Fri Jul 20 14:50:13 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:50:13 +0200 Subject: [pypy-dev] expected_extra_mallocs In-Reply-To: <20070717080253.a1387d0e.simon@arrowtheory.com> References: <20070717080253.a1387d0e.simon@arrowtheory.com> Message-ID: <20070720125013.GB19336@code0.codespeak.net> Hi Simon, On Tue, Jul 17, 2007 at 08:02:53AM -0700, Simon Burton wrote: > def func(): > x = 1. > print x It's a known non-issue: any RPython program calling the "print" statement "leaks" exactly one malloced piece of memory. It's not an issue because in truth the memory is simply kept around to be reused by the next "print". I guess you need to expect one extra malloc in this case. A bientot, Armin. From arigo at tunes.org Fri Jul 20 14:52:54 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:52:54 +0200 Subject: [pypy-dev] rffi strict typing In-Reply-To: <20070719183117.97ad1ce3.simon@arrowtheory.com> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> Message-ID: <20070720125254.GC19336@code0.codespeak.net> Hi Simon, On Thu, Jul 19, 2007 at 06:31:17PM -0700, Simon Burton wrote: > The lltype's are very strict about types. Indeed, it seems worthwhile to loosen this restriction at least for the purpose of calling external functions... Not sure how, but there are possible hacks at least. Armin From fijal at genesilico.pl Sun Jul 22 22:42:57 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sun, 22 Jul 2007 22:42:57 +0200 Subject: [pypy-dev] Strange annotation problem. Message-ID: <46A3C151.5030409@genesilico.pl> Discovered by exarkun. following diff breaks translation :. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: out.diff URL: From amauryfa at gmail.com Mon Jul 23 11:28:52 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 23 Jul 2007 11:28:52 +0200 Subject: [pypy-dev] Strange annotation problem. In-Reply-To: <46A3C151.5030409@genesilico.pl> References: <46A3C151.5030409@genesilico.pl> Message-ID: Hello, Maciek Fijalkowski wrote: > Discovered by exarkun. following diff breaks translation > > :. > > > Index: ../../module/posix/__init__.py > =================================================================== > --- ../../module/posix/__init__.py (revision 45246) > +++ ../../module/posix/__init__.py (working copy) > @@ -83,6 +83,9 @@ > # interpleveldefs['uname'] = 'interp_posix.uname' > if hasattr(os, 'ttyname'): > interpleveldefs['ttyname'] = 'interp_posix.ttyname' > + if hasattr(os, 'setsid'): > + interpleveldefs['xxx'] = 'interp_posix.xxx' > + > for name in w_star: > if hasattr(os, name): > interpleveldefs[name] = 'interp_posix.' + name > Index: ../../module/posix/interp_posix.py > =================================================================== > --- ../../module/posix/interp_posix.py (revision 45246) > +++ ../../module/posix/interp_posix.py (working copy) > @@ -499,6 +499,22 @@ > raise OperationError(space.w_TypeError, space.wrap(msg)) > utime.unwrap_spec = [ObjSpace, str, W_Root] > > +def xxx(space): > + """xxx() -> pid > + > + Stuff > + """ > + return 3 > +xxx.unwrap_spec = [ObjSpace] > + The function seems to return an unwrapped object. does it behave better if you replace the return value with something like space.wrap(3) ? -- Amaury Forgeot d'Arc From fijal at genesilico.pl Mon Jul 23 12:02:29 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 12:02:29 +0200 Subject: [pypy-dev] Strange annotation problem. In-Reply-To: References: <46A3C151.5030409@genesilico.pl> Message-ID: <46A47CB5.2070607@genesilico.pl> Amaury Forgeot d'Arc wrote: > Hello, > > Maciek Fijalkowski wrote: >> Discovered by exarkun. following diff breaks translation >> >> :. >> >> >> Index: ../../module/posix/__init__.py >> =================================================================== >> --- ../../module/posix/__init__.py (revision 45246) >> +++ ../../module/posix/__init__.py (working copy) >> @@ -83,6 +83,9 @@ >> # interpleveldefs['uname'] = 'interp_posix.uname' >> if hasattr(os, 'ttyname'): >> interpleveldefs['ttyname'] = 'interp_posix.ttyname' >> + if hasattr(os, 'setsid'): >> + interpleveldefs['xxx'] = 'interp_posix.xxx' >> + >> for name in w_star: >> if hasattr(os, name): >> interpleveldefs[name] = 'interp_posix.' + name >> Index: ../../module/posix/interp_posix.py >> =================================================================== >> --- ../../module/posix/interp_posix.py (revision 45246) >> +++ ../../module/posix/interp_posix.py (working copy) >> @@ -499,6 +499,22 @@ >> raise OperationError(space.w_TypeError, space.wrap(msg)) >> utime.unwrap_spec = [ObjSpace, str, W_Root] >> >> +def xxx(space): >> + """xxx() -> pid >> + >> + Stuff >> + """ >> + return 3 >> +xxx.unwrap_spec = [ObjSpace] >> + > > The function seems to return an unwrapped object. > does it behave better if you replace the return value with > something like space.wrap(3) ? > Hum. Good point :-) I must be sleeping again... Cheers, fijal :. From anto.cuni at gmail.com Mon Jul 23 13:07:55 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 23 Jul 2007 13:07:55 +0200 Subject: [pypy-dev] Proposal of timeline In-Reply-To: <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> References: <469BE246.20107@gmail.com> <9c0bb8a00707161429v3d8761ag681353aacb7c87d7@mail.gmail.com> <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> Message-ID: <46A48C0B.20705@gmail.com> Hi Paul, I'm cc-ing pypy-dev. Really, don't be shy and don't worry to bother the list with your questions. They are not spam. I'm sure that the others are happy to hear your questions and also to answer in case I can't do it immediately (as happened in the last weekend). Paul deGrandis wrote: > Antonio, > > I've been working on the JVM backend. When I trace the float tests that > look at uint handling, weird NEGATIVE values pop up. I originally > associated these with overflow, but it's a different issue. Any > insight? The way to do unsigned values in Java is to use the next > primary type larger, for example, a uint is stored in a double... but > currently no checks exist for negative values. as Niko pointed out unsigned types are stored in the corresponding signed types in the JVM: this is ok becasue they occupy the same amount of bits, but you have to been careful when dealing with them because the JVM thinks you want to represent a signed value. In particular, if you try to print one to stdout it will be printed as a signed, and so this might be the reason why you see negative values. You have to write a custom procedure to format unsigned values into the right human readable implementation. Please note that this is necessary only during testing or when doing I/O: as long as your unsigned values are kept inside the JVM, they works just fine (apart from comparisons). The simplest way I can think to format an unsigned into an human readable form is to convert it into a double, as I already wrote in a previous email: double from_uint(int x) { double res = x; if (x < 0) // i.e., the leftmost but is 1 res += 4294967294; return res; } But again, you should use this *only* for I/O and testing, not for doing any real operation. > I looked at the test_overflow but these tests don't make sense for the > jvm. You have overflow built in to CLI so you want to make sure that > ovf_check from the rpython definition works, but the JVM defines these > operations. What exactly are these tests aiming to illustrate and test? RPython is a language with a (more or less :-)) well defined semantics, so every backend must take care of implementing this semantics correctly, even if it's not natively supported by the underlying platform. In this particular case, RPython programs expect overflow to be detected correctly. If the JVM does not do this for you, you have to handle those cases by yourself. GenC has the same problem: the C platform does not detect any overflow, so it must explicitly check for that; you can easily reuse the same logic; for example, this macro is taken from translator/c/src/int.h: #define OP_INT_ADD_OVF(x,y,r) \ OP_INT_ADD(x,y,r); \ if ((r^(x)) >= 0 || (r^(y)) >= 0); \ else FAIL_OVF("integer addition") for genjvm, you could e.g. write a java function doing the same (and raising the appropriate exception in case of overflow). > I haven't tackled overriding default values (as skipped in test_class) > yet, but that's really all there is needed left to pass the test_class > requirements I think you could skip this for now. IIRC it's only needed to translate pypy with the thunk objspace, it's not a priority. > I'm still working on the others, but I don't think I'll have trouble > having these first bugs fixed by the end of the work day on the 27th. ok, that's cool! :-) > Lastly, what is the easy approach to I/O and external functions? the easy approach is to forward the external functions to the backend, which will take care of rendering them as appropriate (usually by calling an helper function written in java). This approach is already used to implement time.time, time.clock and os.write: you can find their implementation in jvm/src/pypy/PyPy.java (ll_time_time, ll_time_clock, ll_os_write). The code that calls these functions is in jvm/generator.py, method 'call_primitive'. ciao Anto From fijal at genesilico.pl Mon Jul 23 13:21:41 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 13:21:41 +0200 Subject: [pypy-dev] Proposal of timeline In-Reply-To: <46A48C0B.20705@gmail.com> References: <469BE246.20107@gmail.com> <9c0bb8a00707161429v3d8761ag681353aacb7c87d7@mail.gmail.com> <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> <46A48C0B.20705@gmail.com> Message-ID: <46A48F45.8090505@genesilico.pl> > >> Lastly, what is the easy approach to I/O and external functions? >> > > the easy approach is to forward the external functions to the backend, > which will take care of rendering them as appropriate (usually by > calling an helper function written in java). This approach is already > used to implement time.time, time.clock and os.write: you can find their > implementation in jvm/src/pypy/PyPy.java (ll_time_time, ll_time_clock, > ll_os_write). The code that calls these functions is in > jvm/generator.py, method 'call_primitive'. > > ciao Anto > Small addition from my side. I've been playing recently with a very naive reuse of BasicExternal infrastructure which is supporting external objects in JS backend. Looks nice so far and I'm able to hard code some RPython-accessible objects (in a very ad hoc manner, mind you). I don't know Java that well, but I would be happy to help with this issue. Cheers, fijal :. From amauryfa at gmail.com Mon Jul 23 14:32:46 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 23 Jul 2007 14:32:46 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes Message-ID: Hello again, > Author: fijal > Date: Mon Jul 23 13:52:26 2007 > New Revision: 45269 > > Modified: > pypy/dist/pypy/rpython/module/ll_os.py > pypy/dist/pypy/rpython/module/test/test_posix.py > Log: > Add os.uname. RPython level. segfaults ll2ctypes in some strange way, > I'm not sure I want to know > > > Modified: pypy/dist/pypy/rpython/module/ll_os.py > ============================================================================== > --- pypy/dist/pypy/rpython/module/ll_os.py (original) > +++ pypy/dist/pypy/rpython/module/ll_os.py Mon Jul 23 13:52:26 2007 > @@ -110,6 +110,34 @@ > register_external(os.setsid, [], int, export_name="ll_os.ll_os_setsid", > llimpl=setsid_lltypeimpl) > > +# ------------------------------- os.uname ------------------------------ > + > +if hasattr(os, 'uname'): > + UTSNAMEP = rffi.CStruct('utsname', ('sysname', rffi.CCHARP), > + ('nodename', rffi.CCHARP), > + ('release', rffi.CCHARP), > + ('version', rffi.CCHARP), > + ('machine', rffi.CCHARP), > + ('stuff', rffi.CCHARP)) I think the error comes from the ctypes interpretation of the structure. utsname is not made of char* pointers, but (at least on the debian machine I have access to) the members are actually fixed arrays of chars, the lengths of which are not public, but well specified in sys/utsname.h. It should make no difference when translated, because the code looks like v->sysname etc. ctypes on the other hand needs to know the precise sizeof and offsets of each member. I tried to modify the _socket module to also use rffi, and had the same problem. I think that we will need a tool similar to pypy.rpython.rctypes.tool.ctypes_platform, which generates and compiles C code to get the different sizes and offsets. (We will need support for C defines as well) Hope this helps, -- Amaury Forgeot d'Arc From fijal at genesilico.pl Mon Jul 23 14:36:37 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 14:36:37 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: References: Message-ID: <46A4A0D5.9020405@genesilico.pl> Amaury Forgeot d'Arc wrote: > Hello again, > >> Author: fijal >> Date: Mon Jul 23 13:52:26 2007 >> New Revision: 45269 >> >> Modified: >> pypy/dist/pypy/rpython/module/ll_os.py >> pypy/dist/pypy/rpython/module/test/test_posix.py >> Log: >> Add os.uname. RPython level. segfaults ll2ctypes in some strange way, >> I'm not sure I want to know >> >> >> Modified: pypy/dist/pypy/rpython/module/ll_os.py >> ============================================================================== >> >> --- pypy/dist/pypy/rpython/module/ll_os.py (original) >> +++ pypy/dist/pypy/rpython/module/ll_os.py Mon Jul 23 13:52:26 2007 >> @@ -110,6 +110,34 @@ >> register_external(os.setsid, [], int, >> export_name="ll_os.ll_os_setsid", >> llimpl=setsid_lltypeimpl) >> >> +# ------------------------------- os.uname >> ------------------------------ >> + >> +if hasattr(os, 'uname'): >> + UTSNAMEP = rffi.CStruct('utsname', ('sysname', rffi.CCHARP), >> + ('nodename', rffi.CCHARP), >> + ('release', rffi.CCHARP), >> + ('version', rffi.CCHARP), >> + ('machine', rffi.CCHARP), >> + ('stuff', rffi.CCHARP)) > > I think the error comes from the ctypes interpretation of the > structure. utsname is not made of char* pointers, but (at least on the > debian machine I have access to) the members are actually fixed arrays > of chars, the lengths of which are not public, but well specified in > sys/utsname.h. > > It should make no difference when translated, because the code looks > like v->sysname etc. > ctypes on the other hand needs to know the precise sizeof and offsets > of each member. > > I tried to modify the _socket module to also use rffi, and had the > same problem. > > I think that we will need a tool similar to > pypy.rpython.rctypes.tool.ctypes_platform, which generates and > compiles C code to get the different sizes and offsets. > (We will need support for C defines as well) > > Hope this helps, > I think armin worked a bit on it. We need some more support for that I'm afraid. Let's dig... :. From fijal at genesilico.pl Mon Jul 23 14:38:32 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 14:38:32 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: References: Message-ID: <46A4A148.2090802@genesilico.pl> From the manpage: The length of the arrays in a struct utsname is unspecified; the fields are terminated by a null byte (?? ?). ??? I'm completely confused. :. From arigo at tunes.org Tue Jul 24 18:08:22 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 24 Jul 2007 18:08:22 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <46A4A148.2090802@genesilico.pl> References: <46A4A148.2090802@genesilico.pl> Message-ID: <20070724160822.GA25719@code0.codespeak.net> Hi Fijal, On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: > The length of the arrays in a struct utsname is unspecified; the > fields > are terminated by a null byte (?????? ???). > > I'm completely confused. It means that the structure looks like this in lltype notation: Struct('utsname', ('text1', FixedSizeArray(Char, 10)), ('text2', FixedSizeArray(Char, 20)), ...) where, annoyingly, the numbers '10' and '20' are platform-dependent and must be obtained by trying to compile snippets of C code using sizeof() and offsetof(). Or are you confused about the bit saying that the fields are zero-terminated? It just means that in the array, characters 0 to N-1 are non-null, charater N is null, and the rest of the characters are undefined. A bientot, Armin. From tismer at stackless.com Tue Jul 24 21:18:48 2007 From: tismer at stackless.com (Christian Tismer) Date: Tue, 24 Jul 2007 21:18:48 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <20070724160822.GA25719@code0.codespeak.net> References: <46A4A148.2090802@genesilico.pl> <20070724160822.GA25719@code0.codespeak.net> Message-ID: <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> On 24.07.2007, at 18:08, Armin Rigo wrote: > Hi Fijal, > > On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: >> The length of the arrays in a struct utsname is >> unspecified; the >> fields >> are terminated by a null byte (?????? ???). >> >> I'm completely confused. > > It means that the structure looks like this in lltype notation: > > Struct('utsname', ('text1', FixedSizeArray(Char, 10)), > ('text2', FixedSizeArray(Char, 20)), > ...) > > where, annoyingly, the numbers '10' and '20' are platform-dependent > and > must be obtained by trying to compile snippets of C code using > sizeof() > and offsetof(). > > Or are you confused about the bit saying that the fields are > zero-terminated? It just means that in the array, characters 0 to N-1 > are non-null, charater N is null, and the rest of the characters are > undefined. Which reminds us to implement this carefully and not to rely on the terminating byte, or we will open the door to buffer overflow exploits :-) ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Tue Jul 24 21:33:09 2007 From: tismer at stackless.com (Christian Tismer) Date: Tue, 24 Jul 2007 21:33:09 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <20070717134238.a1c0d9ab.simon@arrowtheory.com> References: <469CB560.9030108@gmx.de> <20070717134238.a1c0d9ab.simon@arrowtheory.com> Message-ID: <1D942D29-1FEB-4228-9EA2-625B295EE659@stackless.com> As Jacob mentioned, I worked on fixing Stackless Pickling. I thought this would be a quick thing for the first morning, but it blocked me very much until the end of the sprint, where a discussion with Samuele clarified to us (we both didn't understand it for quite long). The problem holds for any app-level implementation of any switching concept that should be pickled: Unless you implement this very ugly with global variables, only, you will always have a reference to the tasklet/whatever on the python stack, that tries to do the pickling. I.E. you always end up pickling the pickler itself, in principal. This does not happen from the main tasklet, because it has special handing for that case. This needs to be re-thought. Some special code, somewhat similar to resume points in RPython needs to be implemented, to shortcut the frames on the Python stack which should not be visible (and therefore being pickled) at all. Hoping to be more productive and communicative next time - this was a huge blocker. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijal at genesilico.pl Wed Jul 25 09:17:03 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 25 Jul 2007 09:17:03 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> References: <46A4A148.2090802@genesilico.pl> <20070724160822.GA25719@code0.codespeak.net> <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> Message-ID: <46A6F8EF.9030202@genesilico.pl> Christian Tismer wrote: > > On 24.07.2007, at 18:08, Armin Rigo wrote: > >> Hi Fijal, >> >> On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: >>> The length of the arrays in a struct utsname is unspecified; the >>> fields >>> are terminated by a null byte (?????? ???). >>> >>> I'm completely confused. >> >> It means that the structure looks like this in lltype notation: >> >> Struct('utsname', ('text1', FixedSizeArray(Char, 10)), >> ('text2', FixedSizeArray(Char, 20)), >> ...) >> >> where, annoyingly, the numbers '10' and '20' are platform-dependent and >> must be obtained by trying to compile snippets of C code using sizeof() >> and offsetof(). >> >> Or are you confused about the bit saying that the fields are >> zero-terminated? It just means that in the array, characters 0 to N-1 >> are non-null, charater N is null, and the rest of the characters are >> undefined. > > Which reminds us to implement this carefully and not to rely on the > terminating byte, or we will open the door to buffer overflow exploits :-) > > ciao - chris > I personally don't think that implementation of uname() might be a security hazard - if kernel really wants to hack us it can for sure :] Cheers, fijal :. From fijal at genesilico.pl Thu Jul 26 14:10:32 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Thu, 26 Jul 2007 14:10:32 +0200 Subject: [pypy-dev] Design question Message-ID: <46A88F38.4040009@genesilico.pl> I'm approaching "how to make cache failures not crash on you every time you import anything". So in short, file like ll_os.py looks right now: if cache.defined('XXX'): TYPE = cache.inttype(...) register_external(...) Idea is that if any of the functions involved in creation of external (cache functions...), show a warning and than create a register_external stub which will explode if referenced (by ie a specific extregistry entry). The question is how it should look like than. My idea is: class Os(registeringclass): def register_os_uname(...): ... def register_os_other(...): ... and so on. This will create huge indentation. Any other ideas? Cheers, fijal :. From jgustak at gmail.com Thu Jul 26 17:47:16 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 26 Jul 2007 17:47:16 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: > Next step is to implement macros. We have working simple macros. They are hygienic and referentially transparent. There is no support so you can't use (expr ...) syntax yet. There is also no support for recursive expanding also. There was again some ExecutionContext refactoring. Still only (let ...) and (let* ...) are fully compatible with macro expanding. (letrec ...) is not. In the future: more macros! (ellipsis, recursive expanding and CPS style macros) Cheers, Jakub Gustak From jgustak at gmail.com Thu Jul 26 19:37:59 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 26 Jul 2007 19:37:59 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Some more info. > We have working simple macros. They are hygienic and referentially transparent. r5rs states that if macro introduces new binding, it should be renamed to avoid conflicts with other identifiers. In our case we create syntactic closure like described in: http://people.csail.mit.edu/people/jhbrown/scheme/macroslides04.pdf ExecutionContext.sput() (creating new binding) checks if it deals with syntactic closure or not, and depending on that binds location in proper context. W_Transformer can be called with quoted expression as argument, then appropriate template will expand and then expanded expression will be evaluated. This way of evaluation will surely cause problems with recursive macros, so it is not recommended. __ Jakub From fijal at genesilico.pl Fri Jul 27 23:54:30 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 27 Jul 2007 23:54:30 +0200 Subject: [pypy-dev] partial evaluation links Message-ID: <46AA6996.3070101@genesilico.pl> not sure, if this is of any interest: http://partial-eval.org/pe-impl.html list of partial evaluation implementations (mostly scheme, but also C). Cheers, fijal :. From fijal at genesilico.pl Sun Jul 29 20:30:26 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sun, 29 Jul 2007 20:30:26 +0200 Subject: [pypy-dev] A quick question Message-ID: <46ACDCC2.3080104@genesilico.pl> Can we move jit tests to the end of nightly test run? It'll help quick-checking whether there are no breakages spread all around. Cheers, fijal :.