From fijal at codespeak.net Tue Jul 1 00:03:22 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Tue, 1 Jul 2008 00:03:22 +0200 (CEST) Subject: [pypy-svn] r56196 - pypy/extradoc/talk/ep2008 Message-ID: <20080630220322.BEC30168515@codespeak.net> Author: fijal Date: Tue Jul 1 00:03:22 2008 New Revision: 56196 Modified: pypy/extradoc/talk/ep2008/status.txt Log: small update Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Tue Jul 1 00:03:22 2008 @@ -1,12 +1,19 @@ - +=========== PyPy status =========== +:Author: Maciej Fijalkowski +:Location: Europython 2008 +:Date: 7 July 2008 + +What this talk is about +======================= + * more details recent developements -* where we're going +* our plans -* sponsorship (?) +* how we're going to achieve this Production ready ===================== From fijal at codespeak.net Tue Jul 1 01:01:52 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Tue, 1 Jul 2008 01:01:52 +0200 (CEST) Subject: [pypy-svn] r56197 - pypy/extradoc/talk/ep2008 Message-ID: <20080630230152.A123816A008@codespeak.net> Author: fijal Date: Tue Jul 1 01:01:46 2008 New Revision: 56197 Modified: pypy/extradoc/talk/ep2008/status.txt Log: Update of this talk Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Tue Jul 1 01:01:46 2008 @@ -15,6 +15,8 @@ * how we're going to achieve this +* more or less recap of our blog + Production ready ===================== @@ -23,6 +25,9 @@ * Sometimes requiring to change applications slightly +* Especially refcounting details tend to be a problem + open('xxx', 'w').write('stuff') + Ctypes ====== @@ -35,6 +40,10 @@ * Can handle ie pysqlite-ctypes, pyglet, pymunk or Sole Scion +XXX demo of sole scion and pyglet + +* ctypes is getting better as a side effect + Ctypes configure ================ @@ -46,6 +55,8 @@ * Can handle #defines, types, structure layout etc. +XXX demo + Sqlite ====== @@ -65,13 +76,18 @@ * We'll run 1.0 definitely +XXX demo + Pylons ====== * Worked almost out of the box once eggs are working (1 day) -* No sqlalchemy yet +* No sqlalchemy yet, obscure problems + ahead + +XXX demo Twisted & Nevow =============== @@ -80,14 +96,18 @@ * Nevow works -* We don't support pycrypto nor pyopenssl +* We don't support pycrypto nor pyopenssl and we + won't -Bittorrent -========== +Other software +============== -XXX fill me -http://morepypy.blogspot.com/2008/03/bittorrent-on-pypy.html +* bittorrent +* pypy translation toolchain + +* various smaller things, templating engines, + most pure-python software Conclusion ========== @@ -108,32 +128,60 @@ XXX - relying on untested and undocumented private stuff (zipimport._zip_directory_cache) +XXX - exact strings in exceptions + Speed - comparison with CPython =============================== -XXX fill me +* We're something between 0.8-2x slower than + cpython on various benchmarks. + +* gcbench - 0.8 (fast gc) + +* We're progressing slowly Speed - decent gcs ================== * Faster than refcounting -* Still need some glitches +* Better handling of obscure cases + +* Troubles with ie communication with C -Speed - JIT -=========== +Speed - JIT generator +===================== * Not ready yet! +* Will be super fast + +* Prolog prototype + Other backends ============== * pypy-jvm runs! -* more integration between pypy-cli and .NET +* More integration between pypy-cli and .NET + +* General speed improvements + +* Both backends are progressing - very slowly though + +Plans +===== + +* More JIT - faster Python + +* Support full cpython's stdlib + +* Get more funding money -* general speed improvements +No-plans +======== -* both backends are progressing - very slowly though +* RPython extensions for CPython -XXX anything else? +* Maintain arbitrary number of third-party + extension modules From fijal at codespeak.net Tue Jul 1 01:15:58 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Tue, 1 Jul 2008 01:15:58 +0200 (CEST) Subject: [pypy-svn] r56198 - pypy/extradoc/talk/ep2008 Message-ID: <20080630231558.2212316A0D4@codespeak.net> Author: fijal Date: Tue Jul 1 01:15:57 2008 New Revision: 56198 Modified: pypy/extradoc/talk/ep2008/gc.txt Log: Try to walk around this a bit, I still don't like it too much Modified: pypy/extradoc/talk/ep2008/gc.txt ============================================================================== --- pypy/extradoc/talk/ep2008/gc.txt (original) +++ pypy/extradoc/talk/ep2008/gc.txt Tue Jul 1 01:15:57 2008 @@ -7,22 +7,63 @@ * GC is *not* only the part that takes care of circular references -... - What is refcounting? ==================== -XXX explain +* Store a field on each object which is a number + of references + +* Each time you play with it increase refcount + +* When refcount goes to 0, delete object What is a generational moving GC? ================================= -XXX explain +XXX explain, probably few slides, quick: + +* Different generations of objects + +* Collection runs from roots to all objects + +* Young objects are kept in nursery + +* Older ones are moved away + +* Write barriers, old objects referencing young ones Difference in performance ========================= -XXX +* no mutable fields on objects in generational + (cache friendly) + +* stuff moves around sometimes + +* assuming certain usecases (not many old objects referencing + young ones) + +Allocation cost +================ + +* refcounting/C malloc - expensive + +* generational gc - cheap + +Allocating large amount of objects +================================== + +* CPython - O(n^2) where n is number of objects + allocated + +* Any reasonable generational GC should do better + +Collection costs +================ + +* Full collection - costly (always the case with cpython) + +* Nursery collection - quick Finalizers ========== @@ -35,12 +76,33 @@ Finalizers - resurrection ========================= -XXX +* what happens if your __del__ puts reference back + to itself? + +* in CPython it can be called more than once + +* in other implementations, only once Finalizers - cycles with __del__ ================================ -XXX +* in CPython, they never get collected + +* interesting example - a module global with __del__ + +* in PyPy, cycle gets broken in random order + +* no hacks like globals clearing + +When to call __del__? +===================== + +* example open('x', 'w').write('stuff') + +* on refcounting, flushes file immediately + +* on any other gc, it might be deferred for + a while Calling C-level code ==================== @@ -49,33 +111,9 @@ * Problems with write barriers -... - Strange operation costs ======================= * id XXX what else? - -Allocation cost -================ - -* refcounting/C malloc - expensive - -* generational gc - cheap - -Allocating large amount of objects -================================== - -* CPython - O(n^2) where n is number of objects - allocated - -* Any reasonable generational GC should do better - -Collection costs -================ - -* Full collection - costly (always the case with cpython) - -* Nursery collection - quick From arigo at codespeak.net Tue Jul 1 17:25:01 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 17:25:01 +0200 (CEST) Subject: [pypy-svn] r56200 - pypy/extradoc/talk/ep2008 Message-ID: <20080701152501.597FB16A151@codespeak.net> Author: arigo Date: Tue Jul 1 17:24:59 2008 New Revision: 56200 Modified: pypy/extradoc/talk/ep2008/gc.txt Log: Review attempt, but still not completely sure what we try to say in this talk :-) Modified: pypy/extradoc/talk/ep2008/gc.txt ============================================================================== --- pypy/extradoc/talk/ep2008/gc.txt (original) +++ pypy/extradoc/talk/ep2008/gc.txt Tue Jul 1 17:24:59 2008 @@ -2,6 +2,19 @@ Intro ===== +:: + + From arigo to fijal: should we discuss this a bit? I still don't + know exactly what this talk is trying to achieve but I would + personally imagine that to give an intro to GC techniques we need + pictures with objects as boxes and references as arrows. Also I + think we should go for "CPython" slide, a "Jython / IronPython" + slide, possibly a "IronPython on Mono" slide to introduce Boehm, and + finally several PyPy slides starting from the simple mark-and-sweep + and progressing to moving GCs with some more motivation, finally + reaching generational moving GCs. + + * Memory management model * GC is *not* only the part that takes @@ -54,7 +67,8 @@ ================================== * CPython - O(n^2) where n is number of objects - allocated + allocated (XXX this is a temporary problem more than + a deep issue, it might be fixed soon in CPython) * Any reasonable generational GC should do better @@ -62,6 +76,7 @@ ================ * Full collection - costly (always the case with cpython) + (XXX wrong, CPython uses 3 generations in its cycle finder) * Nursery collection - quick @@ -89,6 +104,10 @@ * in CPython, they never get collected * interesting example - a module global with __del__ + (XXX not sure I follow what you mean here. Python + was designed so that most objects don't have a + reference to the module they come from, but just + a module name, precisely to avoid cycles) * in PyPy, cycle gets broken in random order From arigo at codespeak.net Tue Jul 1 18:00:26 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 18:00:26 +0200 (CEST) Subject: [pypy-svn] r56202 - in pypy/dist/pypy/module/__builtin__: . test Message-ID: <20080701160026.91944169F80@codespeak.net> Author: arigo Date: Tue Jul 1 18:00:25 2008 New Revision: 56202 Modified: pypy/dist/pypy/module/__builtin__/importing.py pypy/dist/pypy/module/__builtin__/test/test_import.py Log: issue378 resolved Test and fix: when importing modules, open the .py files in universal newlines mode. Modified: pypy/dist/pypy/module/__builtin__/importing.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/importing.py (original) +++ pypy/dist/pypy/module/__builtin__/importing.py Tue Jul 1 18:00:25 2008 @@ -91,7 +91,7 @@ e = None if modtype == PYFILE: filename = filepart + ".py" - stream = streamio.open_file_as_stream(filename, "r") + stream = streamio.open_file_as_stream(filename, "rU") else: assert modtype == PYCFILE filename = filepart + ".pyc" Modified: pypy/dist/pypy/module/__builtin__/test/test_import.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_import.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_import.py Tue Jul 1 18:00:25 2008 @@ -57,6 +57,9 @@ "sys.modules[__name__] = pkg_substituted") setuppkg("pkg_substituted", mod='') p = setuppkg("readonly", x='') + p = setuppkg("pkg_univnewlines") + p.join('__init__.py').write('a=5\nb=6\rc="""hello\r\nworld"""\r') + p.join('mod.py').write('a=15\nb=16\rc="""foo\r\nbar"""\r') # create compiled/x.py and a corresponding pyc file p = setuppkg("compiled", x = "x = 84") @@ -258,6 +261,16 @@ import sys assert glob['sys'] is sys + def test_universal_newlines(self): + import pkg_univnewlines + assert pkg_univnewlines.a == 5 + assert pkg_univnewlines.b == 6 + assert pkg_univnewlines.c == "hello\nworld" + from pkg_univnewlines import mod + assert mod.a == 15 + assert mod.b == 16 + assert mod.c == "foo\nbar" + def _getlong(data): x = marshal.dumps(data) return x[-4:] From arigo at codespeak.net Tue Jul 1 18:07:18 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 18:07:18 +0200 (CEST) Subject: [pypy-svn] r56203 - pypy/dist/pypy/module/sys/test Message-ID: <20080701160718.62254169FEA@codespeak.net> Author: arigo Date: Tue Jul 1 18:07:18 2008 New Revision: 56203 Modified: pypy/dist/pypy/module/sys/test/test_sysmodule.py Log: Fix test. Now sys.setcheckinterval() clamps the value to a sane range, and values <= 0 don't make sense. Modified: pypy/dist/pypy/module/sys/test/test_sysmodule.py ============================================================================== --- pypy/dist/pypy/module/sys/test/test_sysmodule.py (original) +++ pypy/dist/pypy/module/sys/test/test_sysmodule.py Tue Jul 1 18:07:18 2008 @@ -263,7 +263,7 @@ def test_setcheckinterval(self): raises(TypeError, sys.setcheckinterval) orig = sys.getcheckinterval() - for n in 0, 100, 120, orig: # orig last to restore starting state + for n in 1, 100, 120, orig: # orig last to restore starting state sys.setcheckinterval(n) assert sys.getcheckinterval() == n From arigo at codespeak.net Tue Jul 1 18:31:34 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 18:31:34 +0200 (CEST) Subject: [pypy-svn] r56204 - pypy/dist/pypy/objspace/std Message-ID: <20080701163134.22B2616A032@codespeak.net> Author: arigo Date: Tue Jul 1 18:31:33 2008 New Revision: 56204 Modified: pypy/dist/pypy/objspace/std/dictmultiobject.py Log: Small bug. Hard to test because it only makes a small performance difference. Modified: pypy/dist/pypy/objspace/std/dictmultiobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/dictmultiobject.py (original) +++ pypy/dist/pypy/objspace/std/dictmultiobject.py Tue Jul 1 18:31:33 2008 @@ -154,7 +154,8 @@ def get(self, w_lookup): space = self.space - if not _is_str(space, w_lookup) and not _is_sane_hash(space, w_lookup): + if not _is_str(space, w_lookup) and not _is_sane_hash(space, + space.type(w_lookup)): # give hash a chance to raise an exception space.hash(w_lookup) return None @@ -177,7 +178,8 @@ def delitem(self, w_key): space = self.space - if not _is_str(space, w_key) and not _is_sane_hash(space, w_key): + if not _is_str(space, w_key) and not _is_sane_hash(space, + space.type(w_key)): # count hash space.hash(w_key) raise KeyError From arigo at codespeak.net Tue Jul 1 19:03:51 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 19:03:51 +0200 (CEST) Subject: [pypy-svn] r56205 - pypy/dist/pypy/objspace/std/test Message-ID: <20080701170351.ACECA16A0F8@codespeak.net> Author: arigo Date: Tue Jul 1 19:03:50 2008 New Revision: 56205 Modified: pypy/dist/pypy/objspace/std/test/test_dictmultiobject.py Log: issue383 unread Test. Modified: pypy/dist/pypy/objspace/std/test/test_dictmultiobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_dictmultiobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_dictmultiobject.py Tue Jul 1 19:03:50 2008 @@ -31,6 +31,16 @@ def test_emptydict_unhashable(self): raises(TypeError, "{}[['x']]") + def test_string_subclass_via_setattr(self): + skip("issue383") + class S(str): + def __hash__(self): + return 123 + s = S("abc") + setattr(s, s, 42) + assert s.__dict__.keys()[0] is s + assert getattr(s, s) == 42 + class TestW_DictSharing(test_dictobject.TestW_DictObject): def setup_class(cls): From arigo at codespeak.net Tue Jul 1 19:29:17 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 19:29:17 +0200 (CEST) Subject: [pypy-svn] r56208 - in pypy/dist/pypy/module/sys: . test Message-ID: <20080701172917.615E22A80D1@codespeak.net> Author: arigo Date: Tue Jul 1 19:29:17 2008 New Revision: 56208 Modified: pypy/dist/pypy/module/sys/test/test_sysmodule.py pypy/dist/pypy/module/sys/vm.py Log: This is really just a workaround for CPython tests, but I try to provide some (very vague) justification. Modified: pypy/dist/pypy/module/sys/test/test_sysmodule.py ============================================================================== --- pypy/dist/pypy/module/sys/test/test_sysmodule.py (original) +++ pypy/dist/pypy/module/sys/test/test_sysmodule.py Tue Jul 1 19:29:17 2008 @@ -263,7 +263,7 @@ def test_setcheckinterval(self): raises(TypeError, sys.setcheckinterval) orig = sys.getcheckinterval() - for n in 1, 100, 120, orig: # orig last to restore starting state + for n in 0, 100, 120, orig: # orig last to restore starting state sys.setcheckinterval(n) assert sys.getcheckinterval() == n Modified: pypy/dist/pypy/module/sys/vm.py ============================================================================== --- pypy/dist/pypy/module/sys/vm.py (original) +++ pypy/dist/pypy/module/sys/vm.py Tue Jul 1 19:29:17 2008 @@ -66,7 +66,15 @@ def getcheckinterval(space): """Return the current check interval; see setcheckinterval().""" - return space.wrap(space.sys.checkinterval) + # xxx to make tests and possibly some obscure apps happy, if the + # checkinterval is set to the minimum possible value (which is 1) we + # return 0. The idea is that according to the CPython docs, <= 0 + # means "check every virtual instruction, maximizing responsiveness + # as well as overhead". + result = space.sys.checkinterval + if result <= 1: + result = 0 + return space.wrap(result) def exc_info(space): """Return the (type, value, traceback) of the most recent exception From arigo at codespeak.net Tue Jul 1 19:37:16 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 19:37:16 +0200 (CEST) Subject: [pypy-svn] r56209 - pypy/dist/lib-python/modified-2.4.1/test Message-ID: <20080701173716.D2CD02A8078@codespeak.net> Author: arigo Date: Tue Jul 1 19:37:15 2008 New Revision: 56209 Modified: pypy/dist/lib-python/modified-2.4.1/test/test_repr.py Log: Make test a little bit more permissive; on top of PyPy the repr is "" instead of "". Modified: pypy/dist/lib-python/modified-2.4.1/test/test_repr.py ============================================================================== --- pypy/dist/lib-python/modified-2.4.1/test/test_repr.py (original) +++ pypy/dist/lib-python/modified-2.4.1/test/test_repr.py Tue Jul 1 19:37:15 2008 @@ -165,7 +165,7 @@ # XXX doesn't test buffers with no b_base or read-write buffers (see # bufferobject.c). The test is fairly incomplete too. Sigh. x = buffer('foo') - self.failUnless(repr(x).startswith(' Author: arigo Date: Tue Jul 1 19:56:16 2008 New Revision: 56211 Modified: pypy/dist/pypy/interpreter/executioncontext.py Log: Minor simplification. Modified: pypy/dist/pypy/interpreter/executioncontext.py ============================================================================== --- pypy/dist/pypy/interpreter/executioncontext.py (original) +++ pypy/dist/pypy/interpreter/executioncontext.py Tue Jul 1 19:56:16 2008 @@ -295,6 +295,7 @@ # force the tick counter to a valid value -- this actually forces # it to reach BYTECODE_COUNTER_OVERFLOW_BIT at the next opcode. ticker = self.get() + ticker &= ~ self.BYTECODE_COUNTER_OVERFLOW_BIT ticker |= self.BYTECODE_COUNTER_MASK self.set(ticker) @@ -309,10 +310,11 @@ ticker = self.get() if ticker & self.BYTECODE_COUNTER_OVERFLOW_BIT: # We must run the periodic actions now, but first - # reset the bytecode counter (the following logic - # works because the BYTECODE_COUNTER_OVERFLOW_BIT - # is currently set) - ticker &= ~ self.BYTECODE_COUNTER_MASK + # reset the bytecode counter (the following line + # works by assuming that we just overflowed the + # counter, i.e. BYTECODE_COUNTER_OVERFLOW_BIT is + # set but none of the BYTECODE_COUNTER_MASK bits + # are). ticker -= ec.space.sys.checkinterval self.set(ticker) for action in periodic_actions: From arigo at codespeak.net Tue Jul 1 19:56:57 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 19:56:57 +0200 (CEST) Subject: [pypy-svn] r56212 - in pypy/dist/pypy: module/marshal/test objspace/std Message-ID: <20080701175657.8B8EC2A8078@codespeak.net> Author: arigo Date: Tue Jul 1 19:56:57 2008 New Revision: 56212 Modified: pypy/dist/pypy/module/marshal/test/make_test_marshal.py pypy/dist/pypy/module/marshal/test/test_marshal.py pypy/dist/pypy/module/marshal/test/test_marshalimpl.py pypy/dist/pypy/objspace/std/marshal_impl.py Log: Fix marshalling of buffers and objects implementing the buffer protocol, following CPython. Modified: pypy/dist/pypy/module/marshal/test/make_test_marshal.py ============================================================================== --- pypy/dist/pypy/module/marshal/test/make_test_marshal.py (original) +++ pypy/dist/pypy/module/marshal/test/make_test_marshal.py Tue Jul 1 19:56:57 2008 @@ -25,8 +25,6 @@ func.func_code scopefunc.func_code u'hello' - buffer(hello) - buffer(u'unicode, too') set() set([1, 2]) frozenset() Modified: pypy/dist/pypy/module/marshal/test/test_marshal.py ============================================================================== --- pypy/dist/pypy/module/marshal/test/test_marshal.py (original) +++ pypy/dist/pypy/module/marshal/test/test_marshal.py Tue Jul 1 19:56:57 2008 @@ -481,44 +481,6 @@ x = marshal.load(f) assert x == case and type(x) is type(case) - def test_buffer_brace_hello_ecarb_(self): - import sys - hello = "he" - hello += "llo" - def func(x): - return lambda y: x+y - scopefunc = func(42) - import marshal, StringIO - case = buffer(hello) - print "case: %-30s func=buffer_brace_hello_ecarb_" % (case, ) - s = marshal.dumps(case) - x = marshal.loads(s) - assert x == case and type(x) is type(case) - f = StringIO.StringIO() - marshal.dump(case, f) - f.seek(0) - x = marshal.load(f) - assert x == case and type(x) is type(case) - - def test_buffer_brace_u_quote_unicode_comma__too_quote__ecarb_(self): - import sys - hello = "he" - hello += "llo" - def func(x): - return lambda y: x+y - scopefunc = func(42) - import marshal, StringIO - case = buffer(u'unicode, too') - print "case: %-30s func=buffer_brace_u_quote_unicode_comma__too_quote__ecarb_" % (case, ) - s = marshal.dumps(case) - x = marshal.loads(s) - assert x == case and type(x) is type(case) - f = StringIO.StringIO() - marshal.dump(case, f) - f.seek(0) - x = marshal.load(f) - assert x == case and type(x) is type(case) - def test_set_brace__ecarb_(self): import sys hello = "he" Modified: pypy/dist/pypy/module/marshal/test/test_marshalimpl.py ============================================================================== --- pypy/dist/pypy/module/marshal/test/test_marshalimpl.py (original) +++ pypy/dist/pypy/module/marshal/test/test_marshalimpl.py Tue Jul 1 19:56:57 2008 @@ -25,3 +25,15 @@ import marshal z = marshal.loads(buffer('i\x02\x00\x00\x00???')) assert z == 2 + + def test_marshal_buffer_object(self): + import marshal + s = marshal.dumps(buffer('foobar')) + t = marshal.loads(s) + assert type(t) is str and t == 'foobar' + + def test_marshal_bufferlike_object(self): + import marshal, array + s = marshal.dumps(array.array('c', 'asd')) + t = marshal.loads(s) + assert type(t) is str and t == 'asd' Modified: pypy/dist/pypy/objspace/std/marshal_impl.py ============================================================================== --- pypy/dist/pypy/objspace/std/marshal_impl.py (original) +++ pypy/dist/pypy/objspace/std/marshal_impl.py Tue Jul 1 19:56:57 2008 @@ -459,25 +459,6 @@ return PyUnicode_DecodeUTF8(space, space.wrap(u.get_str())) register(TYPE_UNICODE, unmarshal_Unicode) -# not directly supported: -def marshal_w_buffer(space, w_buffer, m): - s = space.str_w(space.str(w_buffer)) - m.atom_str(TYPE_UNKNOWN, s) - -handled_by_any.append( ('buffer', marshal_w_buffer) ) - -app = gateway.applevel(r''' - def string_to_buffer(s): - return buffer(s) -''') - -string_to_buffer = app.interphook('string_to_buffer') - -def unmarshal_buffer(space, u, tc): - w_s = space.wrap(u.get_str()) - return string_to_buffer(space, w_s) -register(TYPE_UNKNOWN, unmarshal_buffer) - app = gateway.applevel(r''' def set_to_list(theset): return [item for item in theset] @@ -528,8 +509,19 @@ w_t = space.builtin.get(name) if space.is_true(space.issubtype(w_type, w_t)): func(space, w_obj, m) - break + return + + # any unknown object implementing the buffer protocol is + # accepted and encoded as a plain string + try: + s = space.bufferstr_w(w_obj) + except OperationError, e: + if not e.match(space, space.w_TypeError): + raise else: - raise_exception(space, "unmarshallable object") + m.atom_str(TYPE_STRING, s) + return + + raise_exception(space, "unmarshallable object") register_all(vars()) From bgola at codespeak.net Tue Jul 1 20:00:16 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Tue, 1 Jul 2008 20:00:16 +0200 (CEST) Subject: [pypy-svn] r56213 - pypy/branch/2.5-features/pypy/interpreter/pyparser Message-ID: <20080701180016.EBEA72A8078@codespeak.net> Author: bgola Date: Tue Jul 1 20:00:16 2008 New Revision: 56213 Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/ebnfgrammar.py Log: small fix to match the new Grammar2.5's grammar. See the import_from rule in Grammar2.5 Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/ebnfgrammar.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyparser/ebnfgrammar.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyparser/ebnfgrammar.py Tue Jul 1 20:00:16 2008 @@ -22,7 +22,7 @@ alternative: sequence ( '|' sequence )+ star: '*' | '+' - sequence: (SYMBOL star? | STRING | option | group )+ + sequence: (SYMBOL star? | STRING star? | option | group )+ option: '[' alternative ']' group: '(' alternative ')' star? """ @@ -55,8 +55,8 @@ # group: '(' alternative ')' group = p.Sequence_n( "group", [p.Token_n('TOK_LPAR', '('), alternative, p.Token_n('TOK_RPAR', ')'), star_opt] ) - # sequence: (SYMBOL | STRING | option | group )+ - string = p.Token_n('TOK_STRING') + # sequence: (SYMBOL star? | STRING star? | option | group )+ + string = p.Sequence_n( "string", [p.Token_n('TOK_STRING'), star_opt] ) alt = p.Alternative_n( "sequence_alt", [symbol, string, option, group] ) sequence.args = [ alt ] From arigo at codespeak.net Tue Jul 1 20:02:23 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 20:02:23 +0200 (CEST) Subject: [pypy-svn] r56214 - pypy/dist/pypy/objspace/std/test Message-ID: <20080701180223.59D8F2A8078@codespeak.net> Author: arigo Date: Tue Jul 1 20:02:22 2008 New Revision: 56214 Modified: pypy/dist/pypy/objspace/std/test/test_longobject.py Log: Re-enable this test, probably disabled by accident. Modified: pypy/dist/pypy/objspace/std/test/test_longobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_longobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_longobject.py Tue Jul 1 20:02:22 2008 @@ -97,7 +97,6 @@ assert y < r <= 0 for x in [-1L, 0L, 1L, 2L ** 100 - 1, -2L ** 100 - 1]: for y in [-105566530L, -1L, 1L, 1034522340L]: - continue print "checking division for %s, %s" % (x, y) check_division(x, y) # special case from python tests: From arigo at codespeak.net Tue Jul 1 20:13:16 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 20:13:16 +0200 (CEST) Subject: [pypy-svn] r56215 - pypy/dist/pypy/objspace/std/test Message-ID: <20080701181316.74EE9169E21@codespeak.net> Author: arigo Date: Tue Jul 1 20:13:15 2008 New Revision: 56215 Modified: pypy/dist/pypy/objspace/std/test/test_longobject.py Log: issue384 unread A test crashing on the comparison "0 == (1L<<9999)". Modified: pypy/dist/pypy/objspace/std/test/test_longobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_longobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_longobject.py Tue Jul 1 20:13:15 2008 @@ -59,6 +59,40 @@ a = 31415926L // 10000000L assert a == 3L + def test_compare(self): + skip("in-progress") + BIG = 1L << 9999 + assert 0 == 0L + assert not (0 != 0L) + assert 0L == 0 + assert not (0L != 0) + assert not (0 == BIG) + assert 0 != BIG + assert not (BIG == 0) + assert BIG != 0 + assert not (0L == BIG) + assert 0L != BIG + assert 0 <= 0L + assert not (0 < 0L) + assert 0 <= BIG + assert 0 < BIG + assert not (BIG <= 0) + assert not (BIG < 0) + assert 0L <= 0L + assert not (0L < 0L) + assert 0L <= BIG + assert 0L < BIG + assert not (BIG <= 0L) + assert not (BIG < 0L) + assert not (0 <= -BIG) + assert not (0 < -BIG) + assert -BIG <= 0 + assert -BIG < 0 + assert not (0L <= -BIG) + assert not (0L < -BIG) + assert -BIG <= 0L + assert -BIG < 0L + def test_conversion(self): class long2(long): pass From arigo at codespeak.net Tue Jul 1 21:30:04 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 1 Jul 2008 21:30:04 +0200 (CEST) Subject: [pypy-svn] r56216 - pypy/extradoc/talk/ep2008 Message-ID: <20080701193004.B951A2A8078@codespeak.net> Author: arigo Date: Tue Jul 1 21:30:02 2008 New Revision: 56216 Modified: pypy/extradoc/talk/ep2008/gc.txt Log: More comments for fijal. Modified: pypy/extradoc/talk/ep2008/gc.txt ============================================================================== --- pypy/extradoc/talk/ep2008/gc.txt (original) +++ pypy/extradoc/talk/ep2008/gc.txt Tue Jul 1 21:30:02 2008 @@ -14,6 +14,15 @@ and progressing to moving GCs with some more motivation, finally reaching generational moving GCs. + I could imagine preparing this talk in a completely different + way than with slides with text -- mostly as images. Some of + the text you've written below would go attached to the + corresponding images. + + Still I'm not sure how, or if, we can fill 25 minutes. + Maybe it's ok to go for a 15-20 minutes talk and give + people time for questions and a larger break. + * Memory management model From arigo at codespeak.net Wed Jul 2 11:40:13 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 2 Jul 2008 11:40:13 +0200 (CEST) Subject: [pypy-svn] r56232 - pypy/dist/pypy/rpython/memory Message-ID: <20080702094013.37F6E16A006@codespeak.net> Author: arigo Date: Wed Jul 2 11:40:09 2008 New Revision: 56232 Modified: pypy/dist/pypy/rpython/memory/gctypelayout.py Log: We can share all zero-length arrays in the gc tables. Modified: pypy/dist/pypy/rpython/memory/gctypelayout.py ============================================================================== --- pypy/dist/pypy/rpython/memory/gctypelayout.py (original) +++ pypy/dist/pypy/rpython/memory/gctypelayout.py Wed Jul 2 11:40:09 2008 @@ -232,6 +232,8 @@ return type_id def offsets2table(self, offsets, TYPE): + if len(offsets) == 0: + TYPE = lltype.Void # we can share all zero-length arrays try: return self.offsettable_cache[TYPE] except KeyError: From arigo at codespeak.net Wed Jul 2 12:00:46 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 2 Jul 2008 12:00:46 +0200 (CEST) Subject: [pypy-svn] r56233 - pypy/dist/pypy/rpython/lltypesystem Message-ID: <20080702100046.33EA016A002@codespeak.net> Author: arigo Date: Wed Jul 2 12:00:43 2008 New Revision: 56233 Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py Log: Whack whack whack. This is an attempt to fix the occasional failure of test_ll_thread. The cause (or one of the causes) for the failure is explained in the following comment: # The issue is that if malloc-removal is not performed, we cannot # write a call to 'funcptr(*real_args)' because this needs to read # values from the GC-managed tuple 'real_args', which we cannot do # between before() and after() calls. Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py ============================================================================== --- pypy/dist/pypy/rpython/lltypesystem/rffi.py (original) +++ pypy/dist/pypy/rpython/lltypesystem/rffi.py Wed Jul 2 12:00:43 2008 @@ -102,7 +102,11 @@ invoke_around_handlers = not sandboxsafe unrolling_arg_tps = unrolling_iterable(enumerate(args)) - def wrapper(*args): + + def decode_args(*args): + """Decode the arguments passed to the external function, + automatically normalizing to the exact low-level types. + """ # XXX the next line is a workaround for the annotation bug # shown in rpython.test.test_llann:test_pbctype. Remove it # when the test is fixed... @@ -140,19 +144,15 @@ arg = cast(TARGET, arg) real_args = real_args + (arg,) to_free = to_free + (freeme,) - if invoke_around_handlers: - before = aroundstate.before - after = aroundstate.after - if before: before() - # NB. it is essential that no exception checking occurs after - # the call to before(), because we don't have the GIL any more! - # It is also essential that no GC pointer is alive between now - # and the end of the function, so that the external function - # calls below don't need to be guarded by GC shadow stack logic - # that would crash if not protected by the GIL! - res = funcptr(*real_args) - if invoke_around_handlers: - if after: after() + return real_args, to_free + decode_args._annspecialcase_ = 'specialize:ll' + decode_args._always_inline_ = True + + def decode_result(res, *to_free): + """Decode the result returned by the low-level function, + automatically normalizing it to a common RPython type + and freeing temporary values. + """ for i, TARGET in unrolling_arg_tps: if to_free[i]: lltype.free(to_free[i], flavor='raw') @@ -162,6 +162,48 @@ elif result is UINT: return cast(lltype.Unsigned, res) return res + decode_result._annspecialcase_ = 'specialize:ll' + decode_result._always_inline_ = True + + # The actual wrapper is generated code in order to avoid *args. The + # issue is that if malloc-removal is not performed, we cannot write + # a call to 'funcptr(*real_args)' because this needs to read values + # from the GC-managed tuple 'real_args', which we cannot do between + # before() and after() calls. + + argnames = ', '.join(['a%d' % i for i in range(len(args))]) + if len(args): + unpackargnames = '(' + argnames + ',)' + else: + unpackargnames = '_' + + source = py.code.Source(""" + def wrapper(%(argnames)s): + %(unpackargnames)s, to_free = decode_args(%(argnames)s) + if invoke_around_handlers: + before = aroundstate.before + after = aroundstate.after + if before: before() + # NB. it is essential that no exception checking occurs after + # the call to before(), because we don't have the GIL any more! + # It is also essential that no GC pointer is alive between now + # and the end of the function, so that the external function + # calls below don't need to be guarded by GC shadow stack logic + # that would crash if not protected by the GIL! + res = funcptr(%(argnames)s) + if invoke_around_handlers: + if after: after() + return decode_result(res, *to_free) + """ % locals()) + + miniglobals = {'decode_args': decode_args, + 'invoke_around_handlers': invoke_around_handlers, + 'aroundstate': aroundstate, + 'funcptr': funcptr, + 'decode_result': decode_result, + } + exec source.compile() in miniglobals + wrapper = miniglobals['wrapper'] wrapper._annspecialcase_ = 'specialize:ll' if invoke_around_handlers: # don't inline, as a hack to guarantee that no GC pointer is alive From arigo at codespeak.net Wed Jul 2 12:38:45 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 2 Jul 2008 12:38:45 +0200 (CEST) Subject: [pypy-svn] r56234 - pypy/dist/pypy/rpython/lltypesystem Message-ID: <20080702103845.DD44A16A030@codespeak.net> Author: arigo Date: Wed Jul 2 12:38:44 2008 New Revision: 56234 Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py Log: Whack whack whack even more. Now, looking at the generated C code, we can see that there is really no operation at all in the critical path except: - release GIL - call the C function with arguments that are all prepared earlier - acquire the GIL I should write a test that check that this is really the case, probably by looking at the graph. Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py ============================================================================== --- pypy/dist/pypy/rpython/lltypesystem/rffi.py (original) +++ pypy/dist/pypy/rpython/lltypesystem/rffi.py Wed Jul 2 12:38:44 2008 @@ -101,85 +101,29 @@ # sandboxsafe is a hint for "too-small-ness" (e.g. math functions). invoke_around_handlers = not sandboxsafe - unrolling_arg_tps = unrolling_iterable(enumerate(args)) - - def decode_args(*args): - """Decode the arguments passed to the external function, - automatically normalizing to the exact low-level types. - """ - # XXX the next line is a workaround for the annotation bug - # shown in rpython.test.test_llann:test_pbctype. Remove it - # when the test is fixed... - assert isinstance(lltype.Signed, lltype.Number) - real_args = () - to_free = () - for i, TARGET in unrolling_arg_tps: - arg = args[i] - freeme = None - if TARGET == CCHARP: - if arg is None: - arg = lltype.nullptr(CCHARP.TO) # None => (char*)NULL - freeme = arg - elif isinstance(arg, str): - arg = str2charp(arg) - # XXX leaks if a str2charp() fails with MemoryError - # and was not the first in this function - freeme = arg - elif _isfunctype(TARGET) and not _isllptr(arg): - # XXX pass additional arguments - if invoke_around_handlers: - arg = llhelper(TARGET, _make_wrapper_for(TARGET, arg, - aroundstate)) - else: - arg = llhelper(TARGET, _make_wrapper_for(TARGET, arg)) - else: - SOURCE = lltype.typeOf(arg) - if SOURCE != TARGET: - if TARGET is lltype.Float: - arg = float(arg) - elif ((isinstance(SOURCE, lltype.Number) - or SOURCE is lltype.Bool) - and (isinstance(TARGET, lltype.Number) - or TARGET is lltype.Bool)): - arg = cast(TARGET, arg) - real_args = real_args + (arg,) - to_free = to_free + (freeme,) - return real_args, to_free - decode_args._annspecialcase_ = 'specialize:ll' - decode_args._always_inline_ = True - - def decode_result(res, *to_free): - """Decode the result returned by the low-level function, - automatically normalizing it to a common RPython type - and freeing temporary values. - """ - for i, TARGET in unrolling_arg_tps: - if to_free[i]: - lltype.free(to_free[i], flavor='raw') - if rarithmetic.r_int is not r_int: - if result is INT: - return cast(lltype.Signed, res) - elif result is UINT: - return cast(lltype.Unsigned, res) - return res - decode_result._annspecialcase_ = 'specialize:ll' - decode_result._always_inline_ = True - # The actual wrapper is generated code in order to avoid *args. The # issue is that if malloc-removal is not performed, we cannot write # a call to 'funcptr(*real_args)' because this needs to read values # from the GC-managed tuple 'real_args', which we cannot do between # before() and after() calls. - - argnames = ', '.join(['a%d' % i for i in range(len(args))]) - if len(args): - unpackargnames = '(' + argnames + ',)' + argnames = ', '.join(['a%d' % i for i in range(len(args))]) + realargnames = ', '.join(['r%d' % i for i in range(len(args))]) + decode_lines = [ + 'r%d = _decode_arg(ARGS[%d], a%d, tag_invoke_around_handlers)' + % (i, i, i) for i in range(len(args))] + decode_lines = '; '.join(decode_lines) + free_lines = [ + '_maybe_free_arg(ARGS[%d], a%d, r%d)' + % (i, i, i) for i in range(len(args))] + free_lines = '; '.join(free_lines) + if invoke_around_handlers: + tag_invoke_around_handlers = _tagged_true else: - unpackargnames = '_' + tag_invoke_around_handlers = _tagged_false source = py.code.Source(""" def wrapper(%(argnames)s): - %(unpackargnames)s, to_free = decode_args(%(argnames)s) + %(decode_lines)s if invoke_around_handlers: before = aroundstate.before after = aroundstate.after @@ -190,17 +134,26 @@ # and the end of the function, so that the external function # calls below don't need to be guarded by GC shadow stack logic # that would crash if not protected by the GIL! - res = funcptr(%(argnames)s) + res = funcptr(%(realargnames)s) if invoke_around_handlers: if after: after() - return decode_result(res, *to_free) + %(free_lines)s + return _decode_result(RESULT, res) """ % locals()) - miniglobals = {'decode_args': decode_args, + # xxx workaround workaround: to avoid strange annotation issues, + # make one version of _decode_arg for each wrapper + my_decode_arg = func_with_new_name(_decode_arg, 'decode_arg') + my_maybe_free_arg = func_with_new_name(_maybe_free_arg, 'maybe_free_arg') + miniglobals = {'ARGS': args, + 'RESULT': result, 'invoke_around_handlers': invoke_around_handlers, + 'tag_invoke_around_handlers': tag_invoke_around_handlers, 'aroundstate': aroundstate, 'funcptr': funcptr, - 'decode_result': decode_result, + '_decode_arg': my_decode_arg, + '_maybe_free_arg': my_maybe_free_arg, + '_decode_result': _decode_result, } exec source.compile() in miniglobals wrapper = miniglobals['wrapper'] @@ -216,10 +169,67 @@ return func_with_new_name(wrapper, name) -def _make_wrapper_for(TP, callable, aroundstate=None): +_tagged_false = lambda: None +_tagged_true = lambda: None + +def _decode_arg(TARGET, arg, tag_invoke_around_handlers): + """Decode one of the arguments passed to the external function, + automatically normalizing to the exact low-level type TARGET. + """ + # XXX the next line is a workaround for the annotation bug + # shown in rpython.test.test_llann:test_pbctype. Remove it + # when the test is fixed... + assert isinstance(lltype.Signed, lltype.Number) + if TARGET == CCHARP: + if arg is None: + arg = lltype.nullptr(CCHARP.TO) # None => (char*)NULL + elif isinstance(arg, str): + arg = str2charp(arg) + # XXX leaks if a str2charp() fails with MemoryError + # and was not the first in this function + elif _isfunctype(TARGET) and not _isllptr(arg): + # XXX pass additional arguments + wrapper = _make_wrapper_for(TARGET, arg, tag_invoke_around_handlers) + arg = llhelper(TARGET, wrapper) + else: + SOURCE = lltype.typeOf(arg) + if SOURCE != TARGET: + if TARGET is lltype.Float: + arg = float(arg) + elif ((isinstance(SOURCE, lltype.Number) + or SOURCE is lltype.Bool) + and (isinstance(TARGET, lltype.Number) + or TARGET is lltype.Bool)): + arg = cast(TARGET, arg) + return arg +_decode_arg._annspecialcase_ = 'specialize:ll' +_decode_arg._always_inline_ = True + +def _maybe_free_arg(TARGET, original_arg, decoded_arg): + if TARGET == CCHARP: + if isinstance(original_arg, str): + lltype.free(decoded_arg, flavor='raw') +_maybe_free_arg._annspecialcase_ = 'specialize:ll' +_maybe_free_arg._always_inline_ = True + +def _decode_result(RESULT, res): + """Decode the result returned by the low-level function, + automatically normalizing it to a common RPython type. + """ + if rarithmetic.r_int is not r_int: + if RESULT is INT: + return cast(lltype.Signed, res) + elif RESULT is UINT: + return cast(lltype.Unsigned, res) + return res +_decode_result._annspecialcase_ = 'specialize:ll' +_decode_result._always_inline_ = True + +def _make_wrapper_for(TP, callable, tag_invoke_around_handlers): """ Function creating wrappers for callbacks. Note that this is cheating as we assume constant callbacks and we just memoize wrappers """ + invoke_around_handlers = tag_invoke_around_handlers is _tagged_true if hasattr(callable, '_errorcode_'): errorcode = callable._errorcode_ else: @@ -228,7 +238,7 @@ args = ', '.join(['a%d' % i for i in range(len(TP.TO.ARGS))]) source = py.code.Source(r""" def wrapper(%s): # no *args - no GIL for mallocing the tuple - if aroundstate is not None: + if invoke_around_handlers: before = aroundstate.before after = aroundstate.after else: @@ -258,6 +268,7 @@ miniglobals['Exception'] = Exception miniglobals['os'] = os miniglobals['we_are_translated'] = we_are_translated + miniglobals['aroundstate'] = aroundstate exec source.compile() in miniglobals return miniglobals['wrapper'] _make_wrapper_for._annspecialcase_ = 'specialize:memo' From arigo at codespeak.net Wed Jul 2 13:00:41 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 2 Jul 2008 13:00:41 +0200 (CEST) Subject: [pypy-svn] r56235 - pypy/dist/pypy/rpython/lltypesystem Message-ID: <20080702110041.9CF242A8081@codespeak.net> Author: arigo Date: Wed Jul 2 13:00:37 2008 New Revision: 56235 Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py Log: Revert r56233 and r56234, as it's not helping in the case where one of the incoming arguments is a GC structure itself (e.g. a string). Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py ============================================================================== --- pypy/dist/pypy/rpython/lltypesystem/rffi.py (original) +++ pypy/dist/pypy/rpython/lltypesystem/rffi.py Wed Jul 2 13:00:37 2008 @@ -101,62 +101,67 @@ # sandboxsafe is a hint for "too-small-ness" (e.g. math functions). invoke_around_handlers = not sandboxsafe - # The actual wrapper is generated code in order to avoid *args. The - # issue is that if malloc-removal is not performed, we cannot write - # a call to 'funcptr(*real_args)' because this needs to read values - # from the GC-managed tuple 'real_args', which we cannot do between - # before() and after() calls. - argnames = ', '.join(['a%d' % i for i in range(len(args))]) - realargnames = ', '.join(['r%d' % i for i in range(len(args))]) - decode_lines = [ - 'r%d = _decode_arg(ARGS[%d], a%d, tag_invoke_around_handlers)' - % (i, i, i) for i in range(len(args))] - decode_lines = '; '.join(decode_lines) - free_lines = [ - '_maybe_free_arg(ARGS[%d], a%d, r%d)' - % (i, i, i) for i in range(len(args))] - free_lines = '; '.join(free_lines) - if invoke_around_handlers: - tag_invoke_around_handlers = _tagged_true - else: - tag_invoke_around_handlers = _tagged_false - - source = py.code.Source(""" - def wrapper(%(argnames)s): - %(decode_lines)s - if invoke_around_handlers: - before = aroundstate.before - after = aroundstate.after - if before: before() - # NB. it is essential that no exception checking occurs after - # the call to before(), because we don't have the GIL any more! - # It is also essential that no GC pointer is alive between now - # and the end of the function, so that the external function - # calls below don't need to be guarded by GC shadow stack logic - # that would crash if not protected by the GIL! - res = funcptr(%(realargnames)s) - if invoke_around_handlers: - if after: after() - %(free_lines)s - return _decode_result(RESULT, res) - """ % locals()) - - # xxx workaround workaround: to avoid strange annotation issues, - # make one version of _decode_arg for each wrapper - my_decode_arg = func_with_new_name(_decode_arg, 'decode_arg') - my_maybe_free_arg = func_with_new_name(_maybe_free_arg, 'maybe_free_arg') - miniglobals = {'ARGS': args, - 'RESULT': result, - 'invoke_around_handlers': invoke_around_handlers, - 'tag_invoke_around_handlers': tag_invoke_around_handlers, - 'aroundstate': aroundstate, - 'funcptr': funcptr, - '_decode_arg': my_decode_arg, - '_maybe_free_arg': my_maybe_free_arg, - '_decode_result': _decode_result, - } - exec source.compile() in miniglobals - wrapper = miniglobals['wrapper'] + unrolling_arg_tps = unrolling_iterable(enumerate(args)) + def wrapper(*args): + # XXX the next line is a workaround for the annotation bug + # shown in rpython.test.test_llann:test_pbctype. Remove it + # when the test is fixed... + assert isinstance(lltype.Signed, lltype.Number) + real_args = () + to_free = () + for i, TARGET in unrolling_arg_tps: + arg = args[i] + freeme = None + if TARGET == CCHARP: + if arg is None: + arg = lltype.nullptr(CCHARP.TO) # None => (char*)NULL + freeme = arg + elif isinstance(arg, str): + arg = str2charp(arg) + # XXX leaks if a str2charp() fails with MemoryError + # and was not the first in this function + freeme = arg + elif _isfunctype(TARGET) and not _isllptr(arg): + # XXX pass additional arguments + if invoke_around_handlers: + arg = llhelper(TARGET, _make_wrapper_for(TARGET, arg, + aroundstate)) + else: + arg = llhelper(TARGET, _make_wrapper_for(TARGET, arg)) + else: + SOURCE = lltype.typeOf(arg) + if SOURCE != TARGET: + if TARGET is lltype.Float: + arg = float(arg) + elif ((isinstance(SOURCE, lltype.Number) + or SOURCE is lltype.Bool) + and (isinstance(TARGET, lltype.Number) + or TARGET is lltype.Bool)): + arg = cast(TARGET, arg) + real_args = real_args + (arg,) + to_free = to_free + (freeme,) + if invoke_around_handlers: + before = aroundstate.before + after = aroundstate.after + if before: before() + # NB. it is essential that no exception checking occurs after + # the call to before(), because we don't have the GIL any more! + # It is also essential that no GC pointer is alive between now + # and the end of the function, so that the external function + # calls below don't need to be guarded by GC shadow stack logic + # that would crash if not protected by the GIL! + res = funcptr(*real_args) + if invoke_around_handlers: + if after: after() + for i, TARGET in unrolling_arg_tps: + if to_free[i]: + lltype.free(to_free[i], flavor='raw') + if rarithmetic.r_int is not r_int: + if result is INT: + return cast(lltype.Signed, res) + elif result is UINT: + return cast(lltype.Unsigned, res) + return res wrapper._annspecialcase_ = 'specialize:ll' if invoke_around_handlers: # don't inline, as a hack to guarantee that no GC pointer is alive @@ -169,67 +174,10 @@ return func_with_new_name(wrapper, name) -_tagged_false = lambda: None -_tagged_true = lambda: None - -def _decode_arg(TARGET, arg, tag_invoke_around_handlers): - """Decode one of the arguments passed to the external function, - automatically normalizing to the exact low-level type TARGET. - """ - # XXX the next line is a workaround for the annotation bug - # shown in rpython.test.test_llann:test_pbctype. Remove it - # when the test is fixed... - assert isinstance(lltype.Signed, lltype.Number) - if TARGET == CCHARP: - if arg is None: - arg = lltype.nullptr(CCHARP.TO) # None => (char*)NULL - elif isinstance(arg, str): - arg = str2charp(arg) - # XXX leaks if a str2charp() fails with MemoryError - # and was not the first in this function - elif _isfunctype(TARGET) and not _isllptr(arg): - # XXX pass additional arguments - wrapper = _make_wrapper_for(TARGET, arg, tag_invoke_around_handlers) - arg = llhelper(TARGET, wrapper) - else: - SOURCE = lltype.typeOf(arg) - if SOURCE != TARGET: - if TARGET is lltype.Float: - arg = float(arg) - elif ((isinstance(SOURCE, lltype.Number) - or SOURCE is lltype.Bool) - and (isinstance(TARGET, lltype.Number) - or TARGET is lltype.Bool)): - arg = cast(TARGET, arg) - return arg -_decode_arg._annspecialcase_ = 'specialize:ll' -_decode_arg._always_inline_ = True - -def _maybe_free_arg(TARGET, original_arg, decoded_arg): - if TARGET == CCHARP: - if isinstance(original_arg, str): - lltype.free(decoded_arg, flavor='raw') -_maybe_free_arg._annspecialcase_ = 'specialize:ll' -_maybe_free_arg._always_inline_ = True - -def _decode_result(RESULT, res): - """Decode the result returned by the low-level function, - automatically normalizing it to a common RPython type. - """ - if rarithmetic.r_int is not r_int: - if RESULT is INT: - return cast(lltype.Signed, res) - elif RESULT is UINT: - return cast(lltype.Unsigned, res) - return res -_decode_result._annspecialcase_ = 'specialize:ll' -_decode_result._always_inline_ = True - -def _make_wrapper_for(TP, callable, tag_invoke_around_handlers): +def _make_wrapper_for(TP, callable, aroundstate=None): """ Function creating wrappers for callbacks. Note that this is cheating as we assume constant callbacks and we just memoize wrappers """ - invoke_around_handlers = tag_invoke_around_handlers is _tagged_true if hasattr(callable, '_errorcode_'): errorcode = callable._errorcode_ else: @@ -238,7 +186,7 @@ args = ', '.join(['a%d' % i for i in range(len(TP.TO.ARGS))]) source = py.code.Source(r""" def wrapper(%s): # no *args - no GIL for mallocing the tuple - if invoke_around_handlers: + if aroundstate is not None: before = aroundstate.before after = aroundstate.after else: @@ -268,7 +216,6 @@ miniglobals['Exception'] = Exception miniglobals['os'] = os miniglobals['we_are_translated'] = we_are_translated - miniglobals['aroundstate'] = aroundstate exec source.compile() in miniglobals return miniglobals['wrapper'] _make_wrapper_for._annspecialcase_ = 'specialize:memo' From arigo at codespeak.net Wed Jul 2 13:14:48 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 2 Jul 2008 13:14:48 +0200 (CEST) Subject: [pypy-svn] r56236 - pypy/dist/pypy/rpython/lltypesystem Message-ID: <20080702111448.0DC692A807B@codespeak.net> Author: arigo Date: Wed Jul 2 13:14:47 2008 New Revision: 56236 Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py Log: A better approach: this should guarantee more or less by construction that no gc operation occurs while the GIL is released. See comments. Modified: pypy/dist/pypy/rpython/lltypesystem/rffi.py ============================================================================== --- pypy/dist/pypy/rpython/lltypesystem/rffi.py (original) +++ pypy/dist/pypy/rpython/lltypesystem/rffi.py Wed Jul 2 13:14:47 2008 @@ -101,6 +101,43 @@ # sandboxsafe is a hint for "too-small-ness" (e.g. math functions). invoke_around_handlers = not sandboxsafe + if invoke_around_handlers: + # The around-handlers are releasing the GIL in a threaded pypy. + # We need tons of care to ensure that no GC operation and no + # exception checking occurs while the GIL is released. + + # The actual call is done by this small piece of non-inlinable + # generated code in order to avoid seeing any GC pointer: + # neither '*args' nor the GC objects originally passed in as + # argument to wrapper(), if any (e.g. RPython strings). + + argnames = ', '.join(['a%d' % i for i in range(len(args))]) + source = py.code.Source(""" + def call_external_function(%(argnames)s): + before = aroundstate.before + after = aroundstate.after + if before: before() + # NB. it is essential that no exception checking occurs here! + res = funcptr(%(argnames)s) + if after: after() + return res + """ % locals()) + miniglobals = {'aroundstate': aroundstate, + 'funcptr': funcptr, + } + exec source.compile() in miniglobals + call_external_function = miniglobals['call_external_function'] + call_external_function._dont_inline_ = True + call_external_function._annspecialcase_ = 'specialize:ll' + call_external_function = func_with_new_name(call_external_function, + 'ccall_' + name) + # don't inline, as a hack to guarantee that no GC pointer is alive + # anywhere in call_external_function + else: + # if we don't have to invoke the aroundstate, we can just call + # the low-level function pointer carelessly + call_external_function = funcptr + unrolling_arg_tps = unrolling_iterable(enumerate(args)) def wrapper(*args): # XXX the next line is a workaround for the annotation bug @@ -140,19 +177,7 @@ arg = cast(TARGET, arg) real_args = real_args + (arg,) to_free = to_free + (freeme,) - if invoke_around_handlers: - before = aroundstate.before - after = aroundstate.after - if before: before() - # NB. it is essential that no exception checking occurs after - # the call to before(), because we don't have the GIL any more! - # It is also essential that no GC pointer is alive between now - # and the end of the function, so that the external function - # calls below don't need to be guarded by GC shadow stack logic - # that would crash if not protected by the GIL! - res = funcptr(*real_args) - if invoke_around_handlers: - if after: after() + res = call_external_function(*real_args) for i, TARGET in unrolling_arg_tps: if to_free[i]: lltype.free(to_free[i], flavor='raw') @@ -163,12 +188,7 @@ return cast(lltype.Unsigned, res) return res wrapper._annspecialcase_ = 'specialize:ll' - if invoke_around_handlers: - # don't inline, as a hack to guarantee that no GC pointer is alive - # in the final part of the wrapper - wrapper._dont_inline_ = True - else: - wrapper._always_inline_ = True + wrapper._always_inline_ = True # for debugging, stick ll func ptr to that wrapper._ptr = funcptr From jlg at codespeak.net Thu Jul 3 10:54:35 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Thu, 3 Jul 2008 10:54:35 +0200 (CEST) Subject: [pypy-svn] r56251 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080703085435.6FD39169EF6@codespeak.net> Author: jlg Date: Thu Jul 3 10:54:23 2008 New Revision: 56251 Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt Log: jlg attending post EP2008 sprint Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt ============================================================================== --- pypy/extradoc/sprintinfo/post-ep2008/people.txt (original) +++ pypy/extradoc/sprintinfo/post-ep2008/people.txt Thu Jul 3 10:54:23 2008 @@ -13,6 +13,7 @@ Antonio Cuni 6-13 http://www.jnn.lt/ Holger Krekel 6-13 http://www.litinterp.lt/ Maciej Fijalkowski 6-13 http://www.jnn.lt/ +Jakub Gustak 6-13 http://hospitalityclub.org/ ==================== ============== ============================ People on the following list were present at previous sprints: From stephan at codespeak.net Thu Jul 3 11:18:21 2008 From: stephan at codespeak.net (stephan at codespeak.net) Date: Thu, 3 Jul 2008 11:18:21 +0200 (CEST) Subject: [pypy-svn] r56252 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080703091821.63C7716A020@codespeak.net> Author: stephan Date: Thu Jul 3 11:18:11 2008 New Revision: 56252 Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt Log: added myself to sprint Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt ============================================================================== --- pypy/extradoc/sprintinfo/post-ep2008/people.txt (original) +++ pypy/extradoc/sprintinfo/post-ep2008/people.txt Thu Jul 3 11:18:11 2008 @@ -14,6 +14,7 @@ Holger Krekel 6-13 http://www.litinterp.lt/ Maciej Fijalkowski 6-13 http://www.jnn.lt/ Jakub Gustak 6-13 http://hospitalityclub.org/ +Stephan Diehl 6-13 http://www.litinterp.lt/ ==================== ============== ============================ People on the following list were present at previous sprints: @@ -26,7 +27,6 @@ Carl Friedrich Bolz ? ? Alexander Schremmer ? ? Holger Krekel ? ? -Stephan Diehl ? ? Toon Verwaest ? ? Camillo Bruni ? ? Christian Tismer ? ? From arigo at codespeak.net Thu Jul 3 11:47:01 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 3 Jul 2008 11:47:01 +0200 (CEST) Subject: [pypy-svn] r56253 - in pypy/dist/pypy/module/__builtin__: . test Message-ID: <20080703094701.DECCE168013@codespeak.net> Author: arigo Date: Thu Jul 3 11:46:50 2008 New Revision: 56253 Modified: pypy/dist/pypy/module/__builtin__/app_functional.py pypy/dist/pypy/module/__builtin__/test/test_functional.py Log: issue360 resolved Adapt filter() to use exactly the same logic as CPython. Modified: pypy/dist/pypy/module/__builtin__/app_functional.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/app_functional.py (original) +++ pypy/dist/pypy/module/__builtin__/app_functional.py Thu Jul 3 11:46:50 2008 @@ -68,37 +68,43 @@ else: return res +def filterstring(function, collection, str_type): + if function is None and type(collection) is str_type: + return collection + res = [] + for i in xrange(len(collection)): + c = collection[i] + if function is None or function(c): + if not isinstance(c, str_type): + raise TypeError("can't filter %s to %s: __getitem__ returned different type", str_type.__name__, str_type.__name__) + res.append(c) + return str_type().join(res) + +def filtertuple(function, collection): + if function is None: + function = bool + res = [] + for i in xrange(len(collection)): + c = collection[i] + if function(c): + res.append(c) + return tuple(res) + def filter(function, collection): """construct a list of those elements of collection for which function is True. If function is None, then return the items in the sequence which are True.""" - str_type = None if isinstance(collection, str): - str_type = str + return filterstring(function, collection, str) elif isinstance(collection, unicode): - str_type = unicode + return filterstring(function, collection, unicode) + elif isinstance(collection, tuple): + return filtertuple(function, collection) - if str_type is not None: - if function is None and type(collection) is str_type: - return collection - res = [] - for i in xrange(len(collection)): - c = collection[i] - if function is None or function(c): - if not isinstance(c, str_type): - raise TypeError("can't filter %s to %s: __getitem__ returned different type", str_type.__name__, str_type.__name__) - res.append(c) - return str_type('').join(res) #added '' to make the annotator happy - if function is None: - res = [item for item in collection if item] - else: - res = [item for item in collection if function(item)] - - if isinstance(collection, tuple): - return tuple(res) + return [item for item in collection if item] else: - return res + return [item for item in collection if function(item)] def zip(*collections): """return a list of tuples, where the nth tuple contains every Modified: pypy/dist/pypy/module/__builtin__/test/test_functional.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_functional.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_functional.py Thu Jul 3 11:46:50 2008 @@ -89,6 +89,13 @@ def test_function(self): assert filter(lambda x: x != "a", "a small text") == " smll text" + assert filter(lambda x: x < 20, [3, 33, 5, 55]) == [3, 5] + + def test_filter_tuple_calls_getitem(self): + class T(tuple): + def __getitem__(self, i): + return i * 10 + assert filter(lambda x: x != 20, T("abcd")) == (0, 10, 30) class AppTestXRange: def test_xrange(self): From xoraxax at codespeak.net Thu Jul 3 11:59:25 2008 From: xoraxax at codespeak.net (xoraxax at codespeak.net) Date: Thu, 3 Jul 2008 11:59:25 +0200 (CEST) Subject: [pypy-svn] r56254 - pypy/dist/pypy/doc Message-ID: <20080703095925.AFBF416A05B@codespeak.net> Author: xoraxax Date: Thu Jul 3 11:59:23 2008 New Revision: 56254 Modified: pypy/dist/pypy/doc/getting-started.txt Log: Remove --text option. Modified: pypy/dist/pypy/doc/getting-started.txt ============================================================================== --- pypy/dist/pypy/doc/getting-started.txt (original) +++ pypy/dist/pypy/doc/getting-started.txt Thu Jul 3 11:59:23 2008 @@ -633,14 +633,14 @@ To create a standalone executable using the experimental LLVM_ compiler infrastructure:: - ./translate.py --text --batch --backend=llvm targetpypystandalone.py + ./translate.py --batch --backend=llvm targetpypystandalone.py Translating using the CLI backend +++++++++++++++++++++++++++++++++ To create a standalone .NET executable using the `CLI backend`_:: - ./translate.py --text --batch --backend=cli targetpypystandalone.py + ./translate.py --batch --backend=cli targetpypystandalone.py The executable and all its dependecies will be stored in the ./pypy-cli-data directory. To run pypy.NET, you can run From arigo at codespeak.net Thu Jul 3 12:09:22 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 3 Jul 2008 12:09:22 +0200 (CEST) Subject: [pypy-svn] r56255 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080703100922.9B26E16A05E@codespeak.net> Author: arigo Date: Thu Jul 3 12:09:22 2008 New Revision: 56255 Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt Log: My dates. Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt ============================================================================== --- pypy/extradoc/sprintinfo/post-ep2008/people.txt (original) +++ pypy/extradoc/sprintinfo/post-ep2008/people.txt Thu Jul 3 12:09:22 2008 @@ -15,6 +15,7 @@ Maciej Fijalkowski 6-13 http://www.jnn.lt/ Jakub Gustak 6-13 http://hospitalityclub.org/ Stephan Diehl 6-13 http://www.litinterp.lt/ +Armin Rigo 6-13 Ambassador Hotel ==================== ============== ============================ People on the following list were present at previous sprints: @@ -23,7 +24,6 @@ Name Arrive/Depart Accomodation ==================== ============== ===================== Samuele Pedroni ? ? -Armin Rigo ? ? Carl Friedrich Bolz ? ? Alexander Schremmer ? ? Holger Krekel ? ? From arigo at codespeak.net Thu Jul 3 12:14:50 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 3 Jul 2008 12:14:50 +0200 (CEST) Subject: [pypy-svn] r56256 - pypy/dist/pypy/lang/gameboy Message-ID: <20080703101450.9953116A06B@codespeak.net> Author: arigo Date: Thu Jul 3 12:14:50 2008 New Revision: 56256 Added: pypy/dist/pypy/lang/gameboy/debug.py (contents, props changed) Log: the author forgot to check in this module. this is a dummy template to make the tests pass. Added: pypy/dist/pypy/lang/gameboy/debug.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lang/gameboy/debug.py Thu Jul 3 12:14:50 2008 @@ -0,0 +1,3 @@ +# the author forgot to check in this module +# this is a dummy template to make the tests pass +DEBUG = False From arigo at codespeak.net Thu Jul 3 12:18:57 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 3 Jul 2008 12:18:57 +0200 (CEST) Subject: [pypy-svn] r56257 - in pypy/dist: lib-python/modified-2.4.1 pypy/lib pypy/lib/app_test Message-ID: <20080703101857.A36A616A06A@codespeak.net> Author: arigo Date: Thu Jul 3 12:18:57 2008 New Revision: 56257 Added: pypy/dist/pypy/lib/functools.py - copied unchanged from r56231, pypy/dist/lib-python/modified-2.4.1/functools.py Removed: pypy/dist/lib-python/modified-2.4.1/functools.py Modified: pypy/dist/pypy/lib/app_test/test_functools.py Log: "modified-2.4.1" is only for files copied from "2.4.1" and modified. New files, even if they come from the Python 2.5 svn, should go to pypy/lib for now. In this way we can fix and pass lib/app_test/test_functools. Modified: pypy/dist/pypy/lib/app_test/test_functools.py ============================================================================== --- pypy/dist/pypy/lib/app_test/test_functools.py (original) +++ pypy/dist/pypy/lib/app_test/test_functools.py Thu Jul 3 12:18:57 2008 @@ -1,4 +1,4 @@ -import functools +from pypy.lib import functools import unittest from test import test_support from weakref import proxy From arigo at codespeak.net Thu Jul 3 14:34:41 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 3 Jul 2008 14:34:41 +0200 (CEST) Subject: [pypy-svn] r56258 - pypy/dist/pypy/objspace/std/test Message-ID: <20080703123441.C56992D8001@codespeak.net> Author: arigo Date: Thu Jul 3 14:34:40 2008 New Revision: 56258 Modified: pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py Log: A skipped test that shows an obscure failure triggered by the builtinshortcut logic :-( Modified: pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py (original) +++ pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py Thu Jul 3 14:34:40 2008 @@ -1,7 +1,23 @@ from pypy.objspace.std.test import test_userobject +WITH_BUILTINSHORTCUT = {'objspace.std.builtinshortcut': True} + class AppTestUserObject(test_userobject.AppTestUserObject): - OPTIONS = {'objspace.std.builtinshortcut': True} + OPTIONS = WITH_BUILTINSHORTCUT class AppTestWithMultiMethodVersion2(test_userobject.AppTestWithMultiMethodVersion2): - OPTIONS = {'objspace.std.builtinshortcut': True} + OPTIONS = WITH_BUILTINSHORTCUT + +class AppTestBug: + def setup_class(cls): + from pypy import conftest + cls.space = conftest.gettestobjspace(**WITH_BUILTINSHORTCUT) + + def test_frozen_subtype(self): + skip("in-progress") + class S(set): pass + assert S("abc") == set("abc") + assert set("abc") == S("abc") + class F(frozenset): pass + assert F("abc") == frozenset("abc") + assert frozenset("abc") == F("abc") From arigo at codespeak.net Thu Jul 3 15:38:15 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 3 Jul 2008 15:38:15 +0200 (CEST) Subject: [pypy-svn] r56259 - in pypy/dist/pypy: annotation rpython rpython/lltypesystem rpython/ootypesystem rpython/test translator/backendopt translator/cli translator/cli/test translator/jvm translator/jvm/src/pypy translator/jvm/test translator/oosupport/test_template Message-ID: <20080703133815.11030168568@codespeak.net> Author: arigo Date: Thu Jul 3 15:38:13 2008 New Revision: 56259 Added: pypy/dist/pypy/translator/jvm/src/pypy/PyPyThrowable.java - copied unchanged from r56130, pypy/branch/less-meta-instances/pypy/translator/jvm/src/pypy/PyPyThrowable.java pypy/dist/pypy/translator/oosupport/test_template/exception.py - copied unchanged from r56130, pypy/branch/less-meta-instances/pypy/translator/oosupport/test_template/exception.py Modified: pypy/dist/pypy/annotation/description.py pypy/dist/pypy/annotation/model.py pypy/dist/pypy/rpython/llinterp.py pypy/dist/pypy/rpython/lltypesystem/rclass.py pypy/dist/pypy/rpython/lltypesystem/rpbc.py pypy/dist/pypy/rpython/ootypesystem/exceptiondata.py pypy/dist/pypy/rpython/ootypesystem/ootype.py pypy/dist/pypy/rpython/ootypesystem/rbuiltin.py pypy/dist/pypy/rpython/ootypesystem/rclass.py pypy/dist/pypy/rpython/ootypesystem/rootype.py pypy/dist/pypy/rpython/ootypesystem/rpbc.py pypy/dist/pypy/rpython/rclass.py pypy/dist/pypy/rpython/rpbc.py pypy/dist/pypy/rpython/test/test_normalizecalls.py pypy/dist/pypy/rpython/test/test_rbuiltin.py pypy/dist/pypy/rpython/test/test_rclass.py pypy/dist/pypy/translator/backendopt/inline.py pypy/dist/pypy/translator/cli/database.py pypy/dist/pypy/translator/cli/function.py pypy/dist/pypy/translator/cli/opcodes.py pypy/dist/pypy/translator/cli/test/runtest.py pypy/dist/pypy/translator/cli/test/test_exception.py pypy/dist/pypy/translator/jvm/database.py pypy/dist/pypy/translator/jvm/node.py pypy/dist/pypy/translator/jvm/opcodes.py pypy/dist/pypy/translator/jvm/test/test_exception.py pypy/dist/pypy/translator/jvm/typesystem.py Log: (antocuni, niko, arigo) Merge the less-meta-instances branch. Now ootypesystem.rclass.OBJECT is empty. We removed the "meta" field and only create meta-classes and meta-instances when necessary instead of systematically. This lets us improve the oo backends too. In the future this might also let us map exceptions more directly to native ones, possibly. svn merge svn+ssh://codespeak.net/svn/pypy/branch/less-meta-instances at 53334 \ svn+ssh://codespeak.net/svn/pypy/branch/less-meta-instances at 56130 Modified: pypy/dist/pypy/annotation/description.py ============================================================================== --- pypy/dist/pypy/annotation/description.py (original) +++ pypy/dist/pypy/annotation/description.py Thu Jul 3 15:38:13 2008 @@ -466,6 +466,9 @@ for name, value in cls.__dict__.items(): self.add_source_attribute(name, value, mixin) + def getallclassdefs(self): + return self._classdefs.values() + def getclassdef(self, key): try: return self._classdefs[key] Modified: pypy/dist/pypy/annotation/model.py ============================================================================== --- pypy/dist/pypy/annotation/model.py (original) +++ pypy/dist/pypy/annotation/model.py Thu Jul 3 15:38:13 2008 @@ -591,6 +591,8 @@ return s_val.ootype if isinstance(s_val, SomeOOStaticMeth): return s_val.method + if isinstance(s_val, SomeOOClass): + return ootype.Class if isinstance(s_val, SomeInteriorPtr): p = s_val.ll_ptrtype if 0 in p.offsets: Modified: pypy/dist/pypy/rpython/llinterp.py ============================================================================== --- pypy/dist/pypy/rpython/llinterp.py (original) +++ pypy/dist/pypy/rpython/llinterp.py Thu Jul 3 15:38:13 2008 @@ -37,7 +37,7 @@ return ''.join(etype.name).rstrip('\x00') else: # ootype! - return etype.class_._INSTANCE._name.split(".")[-1] + return etype._INSTANCE._name.split(".")[-1] class LLInterpreter(object): """ low level interpreter working with concrete values. """ Modified: pypy/dist/pypy/rpython/lltypesystem/rclass.py ============================================================================== --- pypy/dist/pypy/rpython/lltypesystem/rclass.py (original) +++ pypy/dist/pypy/rpython/lltypesystem/rclass.py Thu Jul 3 15:38:13 2008 @@ -166,7 +166,10 @@ if cast_to_typeptr: vtable = cast_vtable_to_typeptr(vtable) return vtable - getruntime = getvtable + + def getruntime(self, expected_type): + assert expected_type == CLASSTYPE + return self.getvtable() def setup_vtable(self, vtable, rsubcls): """Initialize the 'self' portion of the 'vtable' belonging to the Modified: pypy/dist/pypy/rpython/lltypesystem/rpbc.py ============================================================================== --- pypy/dist/pypy/rpython/lltypesystem/rpbc.py (original) +++ pypy/dist/pypy/rpython/lltypesystem/rpbc.py Thu Jul 3 15:38:13 2008 @@ -396,6 +396,8 @@ v_inst1 = hop.gendirectcall(ll_instantiate, vtypeptr) return hop.genop('cast_pointer', [v_inst1], resulttype = r_instance) + def getlowleveltype(self): + return rclass.CLASSTYPE # ____________________________________________________________ Modified: pypy/dist/pypy/rpython/ootypesystem/exceptiondata.py ============================================================================== --- pypy/dist/pypy/rpython/ootypesystem/exceptiondata.py (original) +++ pypy/dist/pypy/rpython/ootypesystem/exceptiondata.py Thu Jul 3 15:38:13 2008 @@ -26,8 +26,8 @@ self.fn_raise_OSError = self.make_raise_OSError(rtyper) def make_exception_matcher(self, rtyper): - # ll_exception_matcher(real_exception_meta, match_exception_meta) - s_classtype = annmodel.SomeOOInstance(self.lltype_of_exception_type) + # ll_exception_matcher(real_exception_class, match_exception_class) + s_classtype = annmodel.SomeOOClass(ootype.ROOT) helper_fn = rtyper.annotate_helper_fn(rclass.ll_issubclass, [s_classtype, s_classtype]) return helper_fn @@ -63,7 +63,6 @@ r_class = rclass.getclassrepr(rtyper, None) r_class.setup() default_excinst = ootype.new(self.lltype_of_exception_value) - default_excinst.meta = r_class.get_meta_instance() # build the table in order base classes first, subclasses last sortedtable = [] Modified: pypy/dist/pypy/rpython/ootypesystem/ootype.py ============================================================================== --- pypy/dist/pypy/rpython/ootypesystem/ootype.py (original) +++ pypy/dist/pypy/rpython/ootypesystem/ootype.py Thu Jul 3 15:38:13 2008 @@ -47,7 +47,10 @@ def _defl(self): return nullruntimeclass - + + def _example(self): + return _class(ROOT) + Class = Class() class Instance(OOType): @@ -780,6 +783,7 @@ self._INSTANCE = INSTANCE nullruntimeclass = _class(None) +Class._null = nullruntimeclass class _instance(object): Modified: pypy/dist/pypy/rpython/ootypesystem/rbuiltin.py ============================================================================== --- pypy/dist/pypy/rpython/ootypesystem/rbuiltin.py (original) +++ pypy/dist/pypy/rpython/ootypesystem/rbuiltin.py Thu Jul 3 15:38:13 2008 @@ -75,15 +75,14 @@ v_obj, v_cls = hop.inputargs(instance_repr, class_repr) if isinstance(v_cls, Constant): - c_cls = hop.inputconst(ootype.Void, v_cls.value.class_._INSTANCE) + c_cls = hop.inputconst(ootype.Void, v_cls.value._INSTANCE) return hop.genop('instanceof', [v_obj, c_cls], resulttype=ootype.Bool) else: return hop.gendirectcall(ll_isinstance, v_obj, v_cls) -def ll_isinstance(inst, meta): - c1 = inst.meta.class_ - c2 = meta.class_ - return ootype.subclassof(c1, c2) +def ll_isinstance(inst, class_): + c1 = ootype.classof(inst) + return ootype.subclassof(c1, class_) def rtype_instantiate(hop): if hop.args_s[0].is_constant(): @@ -99,17 +98,12 @@ else: r_instance = hop.s_result.rtyper_makerepr(hop.rtyper) INSTANCE = r_instance.lowleveltype - c_instance = hop.inputconst(ootype.Void, INSTANCE) - v_cls = hop.inputarg(hop.args_r[0], arg=0) - v_obj = hop.gendirectcall(ll_instantiate, c_instance, v_cls) + class_repr = rclass.get_type_repr(hop.rtyper) + v_cls = hop.inputarg(class_repr, arg=0) + v_obj = hop.genop('runtimenew', [v_cls], resulttype=ootype.ROOT) v_instance = hop.genop('oodowncast', [v_obj], resulttype=hop.r_result.lowleveltype) - c_meta = hop.inputconst(ootype.Void, "meta") - hop.genop("oosetfield", [v_instance, c_meta, v_cls], resulttype=ootype.Void) return v_instance -def ll_instantiate(INST, C): - return ootype.runtimenew(C.class_) - BUILTIN_TYPER = {} BUILTIN_TYPER[ootype.new] = rtype_new BUILTIN_TYPER[ootype.oonewarray] = rtype_oonewarray Modified: pypy/dist/pypy/rpython/ootypesystem/rclass.py ============================================================================== --- pypy/dist/pypy/rpython/ootypesystem/rclass.py (original) +++ pypy/dist/pypy/rpython/ootypesystem/rclass.py Thu Jul 3 15:38:13 2008 @@ -7,49 +7,52 @@ from pypy.rpython.rclass import AbstractClassRepr, AbstractInstanceRepr, \ getinstancerepr, getclassrepr, get_type_repr from pypy.rpython.ootypesystem import ootype +from pypy.rpython.exceptiondata import standardexceptions from pypy.tool.pairtype import pairtype from pypy.tool.sourcetools import func_with_new_name -CLASSTYPE = ootype.Instance("Object_meta", ootype.ROOT, - fields={"class_": ootype.Class}) -OBJECT = ootype.Instance("Object", ootype.ROOT, - fields={'meta': CLASSTYPE}) +OBJECT = ootype.ROOT +META = ootype.Instance("Meta", ootype.ROOT, + fields={"class_": ootype.Class}) class ClassRepr(AbstractClassRepr): def __init__(self, rtyper, classdef): AbstractClassRepr.__init__(self, rtyper, classdef) - + # This is the Repr for a reference to the class 'classdef' or + # any subclass. In the simple case, the lowleveltype is just + # ootype.Class. If we need to store class attributes, we use a + # "meta" class where the attributes are defined, and the class + # reference is a reference to an instance of this meta class. + extra_access_sets = self.rtyper.class_pbc_attributes.get( + classdef, {}) + has_class_attributes = bool(extra_access_sets) if self.classdef is not None: self.rbase = getclassrepr(self.rtyper, self.classdef.basedef) - base_type = self.rbase.lowleveltype - self.lowleveltype = ootype.Instance( - self.classdef.name + "_meta", base_type) + meta_base_type = self.rbase.lowleveltype + baseclass_has_meta = meta_base_type != ootype.Class + else: + baseclass_has_meta = False + + if not has_class_attributes and not baseclass_has_meta: + self.lowleveltype = ootype.Class # simple case else: - # we are ROOT - self.lowleveltype = CLASSTYPE + if self.classdef is None: + raise TyperError("the root 'object' class should not have" + " class attributes") + if self.classdef.classdesc.pyobj in standardexceptions: + raise TyperError("Standard exception class %r should not have" + " class attributes" % (self.classdef.name,)) + if not baseclass_has_meta: + meta_base_type = META + self.lowleveltype = ootype.Instance( + self.classdef.name + "_meta", meta_base_type) def _setup_repr(self): - clsfields = {} pbcfields = {} - if self.classdef is not None: + if self.lowleveltype != ootype.Class: # class attributes llfields = [] - """ - attrs = self.classdef.attrs.items() - attrs.sort() - for name, attrdef in attrs: - if attrdef.readonly: - s_value = attrdef.s_value - s_unboundmethod = self.prepare_method(s_value) - if s_unboundmethod is not None: - allmethods[name] = True - s_value = s_unboundmethod - r = self.rtyper.getrepr(s_value) - mangled_name = 'cls_' + name - clsfields[name] = mangled_name, r - llfields.append((mangled_name, r.lowleveltype)) - """ # attributes showing up in getattrs done on the class as a PBC extra_access_sets = self.rtyper.class_pbc_attributes.get( self.classdef, {}) @@ -61,24 +64,26 @@ self.rbase.setup() ootype.addFields(self.lowleveltype, dict(llfields)) - #self.clsfields = clsfields self.pbcfields = pbcfields self.meta_instance = None def get_meta_instance(self, cast_to_root_meta=True): + if self.lowleveltype == ootype.Class: + raise TyperError("no meta-instance for class %r" % + (self.classdef,)) if self.meta_instance is None: self.meta_instance = ootype.new(self.lowleveltype) self.setup_meta_instance(self.meta_instance, self) meta_instance = self.meta_instance if cast_to_root_meta: - meta_instance = ootype.ooupcast(CLASSTYPE, meta_instance) + meta_instance = ootype.ooupcast(META, meta_instance) return meta_instance def setup_meta_instance(self, meta_instance, rsubcls): if self.classdef is None: rinstance = getinstancerepr(self.rtyper, rsubcls.classdef) - setattr(meta_instance, 'class_', rinstance.lowleveltype._class) + meta_instance.class_ = ootype.runtimeClass(rinstance.lowleveltype) else: # setup class attributes: for each attribute name at the level # of 'self', look up its value in the subclass rsubcls @@ -88,14 +93,6 @@ llvalue = r.convert_desc_or_const(value) setattr(meta_instance, mangled_name, llvalue) - #mro = list(rsubcls.classdef.getmro()) - #for fldname in self.clsfields: - # mangled_name, r = self.clsfields[fldname] - # if r.lowleveltype is Void: - # continue - # value = rsubcls.classdef.classdesc.read_attribute(fldname, None) - # if value is not None: - # assign(mangled_name, value) # extra PBC attributes for (access_set, attr), (mangled_name, r) in self.pbcfields.items(): if rsubcls.classdef.classdesc not in access_set.descs: @@ -107,15 +104,27 @@ assign(mangled_name, attrvalue) # then initialize the 'super' portion of the vtable - meta_instance_super = ootype.ooupcast( - self.rbase.lowleveltype, meta_instance) - self.rbase.setup_meta_instance(meta_instance_super, rsubcls) + self.rbase.setup_meta_instance(meta_instance, rsubcls) + + def getruntime(self, expected_type): + if expected_type == ootype.Class: + rinstance = getinstancerepr(self.rtyper, self.classdef) + return ootype.runtimeClass(rinstance.lowleveltype) + else: + assert ootype.isSubclass(expected_type, META) + meta = self.get_meta_instance(cast_to_root_meta=False) + return ootype.ooupcast(expected_type, meta) - getruntime = get_meta_instance - def fromclasstype(self, vclass, llops): - return llops.genop('oodowncast', [vclass], - resulttype=self.lowleveltype) + assert ootype.isSubclass(vclass.concretetype, META) + if self.lowleveltype == ootype.Class: + c_class_ = inputconst(ootype.Void, 'class_') + return llops.genop('oogetfield', [vclass, c_class_], + resulttype=ootype.Class) + else: + assert ootype.isSubclass(self.lowleveltype, vclass.concretetype) + return llops.genop('oodowncast', [vclass], + resulttype=self.lowleveltype) def getpbcfield(self, vcls, access_set, attr, llops): if (access_set, attr) not in self.pbcfields: @@ -127,12 +136,11 @@ def rtype_issubtype(self, hop): class_repr = get_type_repr(self.rtyper) - vmeta1, vmeta2 = hop.inputargs(class_repr, class_repr) - return hop.gendirectcall(ll_issubclass, vmeta1, vmeta2) + vcls1, vcls2 = hop.inputargs(class_repr, class_repr) + return hop.genop('subclassof', [vcls1, vcls2], resulttype=ootype.Bool) -def ll_issubclass(meta1, meta2): - class1 = meta1.class_ - class2 = meta2.class_ +def ll_issubclass(class1, class2): + # helper for exceptiondata.py return ootype.subclassof(class1, class2) # ____________________________________________________________ @@ -237,7 +245,7 @@ mangled = mangle(meth_name, self.rtyper.getconfig()) allmethods[mangled] = meth_name, s_meth # else: it's the __init__ of a builtin exception - + # # hash() support if self.rtyper.needs_hash_support(self.classdef): @@ -255,6 +263,12 @@ baseInstance = self.lowleveltype._superclass classrepr = getclassrepr(self.rtyper, self.classdef) + # if this class has a corresponding metaclass, attach + # a getmeta() method to get the corresponding meta_instance + if classrepr.lowleveltype != ootype.Class: + oovalue = classrepr.get_meta_instance() + self.attach_class_attr_accessor('getmeta', oovalue) + for mangled, (name, s_value) in allmethods.iteritems(): methdescs = s_value.descriptions origin = dict([(methdesc.originclassdef, methdesc) for @@ -329,7 +343,12 @@ for mangled, (s_value, value) in self.classattributes.items(): r = self.rtyper.getrepr(s_value) - m = self.attach_class_attr_accessor(mangled, value, r) + if value is None: + self.attach_abstract_class_attr_accessor(mangled, + r.lowleveltype) + else: + oovalue = r.convert_desc_or_const(value) + self.attach_class_attr_accessor(mangled, oovalue) # step 4: do the same with instance fields whose default # values are overridden in subclasses. Not sure it's the best @@ -359,21 +378,21 @@ ootype.overrideDefaultForFields(self.lowleveltype, overridden_defaults) - def attach_class_attr_accessor(self, mangled, value, r_value): + def attach_abstract_class_attr_accessor(self, mangled, attrtype): + M = ootype.Meth([], attrtype) + m = ootype.meth(M, _name=mangled, abstract=True) + ootype.addMethods(self.lowleveltype, {mangled: m}) + + def attach_class_attr_accessor(self, mangled, oovalue): def ll_getclassattr(self): return oovalue - M = ootype.Meth([], r_value.lowleveltype) - if value is None: - m = ootype.meth(M, _name=mangled, abstract=True) - else: - oovalue = r_value.convert_desc_or_const(value) - ll_getclassattr = func_with_new_name(ll_getclassattr, - 'll_get_' + mangled) - graph = self.rtyper.annotate_helper(ll_getclassattr, [self.lowleveltype]) - m = ootype.meth(M, _name=mangled, _callable=ll_getclassattr, - graph=graph) - + M = ootype.Meth([], ootype.typeOf(oovalue)) + ll_getclassattr = func_with_new_name(ll_getclassattr, + 'll_get_' + mangled) + graph = self.rtyper.annotate_helper(ll_getclassattr, [self.lowleveltype]) + m = ootype.meth(M, _name=mangled, _callable=ll_getclassattr, + graph=graph) ootype.addMethods(self.lowleveltype, {mangled: m}) def get_ll_hash_function(self): @@ -406,14 +425,21 @@ return hop.genop("oosend", [cname, v_inst], resulttype = hop.r_result.lowleveltype) elif attr == '__class__': - if hop.r_result.lowleveltype is ootype.Void: + expected_type = hop.r_result.lowleveltype + if expected_type is ootype.Void: # special case for when the result of '.__class__' is constant [desc] = hop.s_result.descriptions return hop.inputconst(ootype.Void, desc.pyobj) + elif expected_type == ootype.Class: + return hop.genop('classof', [v_inst], + resulttype = ootype.Class) else: - cmeta = inputconst(ootype.Void, "meta") - return hop.genop('oogetfield', [v_inst, cmeta], - resulttype=CLASSTYPE) + assert expected_type == META + _, meth = v_inst.concretetype._lookup('getmeta') + assert meth + c_getmeta = hop.inputconst(ootype.Void, 'getmeta') + return hop.genop('oosend', [c_getmeta, v_inst], + resulttype = META) else: raise TyperError("no attribute %r on %r" % (attr, self)) @@ -448,8 +474,7 @@ if hop.args_s[0].can_be_none(): return hop.gendirectcall(ll_inst_type, vinst) else: - cmeta = inputconst(ootype.Void, "meta") - return hop.genop('oogetfield', [vinst, cmeta], resulttype=CLASSTYPE) + return hop.genop('classof', [vinst], resulttype=ootype.Class) def null_instance(self): return ootype.null(self.lowleveltype) @@ -465,10 +490,6 @@ classrepr = getclassrepr(self.rtyper, self.classdef) v_instance = llops.genop("new", [inputconst(ootype.Void, self.lowleveltype)], self.lowleveltype) - cmeta = inputconst(ootype.Void, "meta") - cmeta_instance = inputconst(CLASSTYPE, classrepr.get_meta_instance()) - llops.genop("oosetfield", [v_instance, cmeta, cmeta_instance], - resulttype=ootype.Void) return v_instance def initialize_prebuilt_data(self, value, classdef, result): @@ -545,7 +566,7 @@ def ll_inst_type(obj): if obj: - return obj.meta + return ootype.classof(obj) else: # type(None) -> NULL (for now) - return ootype.null(CLASSTYPE) + return ootype.nullruntimeclass Modified: pypy/dist/pypy/rpython/ootypesystem/rootype.py ============================================================================== --- pypy/dist/pypy/rpython/ootypesystem/rootype.py (original) +++ pypy/dist/pypy/rpython/ootypesystem/rootype.py Thu Jul 3 15:38:13 2008 @@ -32,6 +32,11 @@ class OOClassRepr(Repr): lowleveltype = Class + + def rtype_is_true(self, hop): + vlist = hop.inputargs(self) + return hop.genop('oononnull', vlist, resulttype=ootype.Bool) + ooclass_repr = OOClassRepr() class OOInstanceRepr(Repr): Modified: pypy/dist/pypy/rpython/ootypesystem/rpbc.py ============================================================================== --- pypy/dist/pypy/rpython/ootypesystem/rpbc.py (original) +++ pypy/dist/pypy/rpython/ootypesystem/rpbc.py Thu Jul 3 15:38:13 2008 @@ -4,11 +4,12 @@ AbstractFunctionsPBCRepr, AbstractMultipleUnrelatedFrozenPBCRepr, \ none_frozen_pbc_repr from pypy.rpython.rclass import rtype_new_instance, getinstancerepr +from pypy.rpython.rclass import getclassrepr, get_type_repr from pypy.rpython.rpbc import get_concrete_calltable from pypy.rpython import callparse from pypy.rpython.ootypesystem import ootype from pypy.rpython.ootypesystem.rclass import ClassRepr, InstanceRepr -from pypy.rpython.ootypesystem.rclass import mangle +from pypy.rpython.ootypesystem.rclass import mangle, META from pypy.annotation import model as annmodel from pypy.annotation import description from pypy.tool.pairtype import pairtype @@ -42,17 +43,24 @@ class ClassesPBCRepr(AbstractClassesPBCRepr): - def _instantiate_runtime_class(self, hop, v_meta, r_instance): + def _instantiate_runtime_class(self, hop, v_class, r_instance): classdef = hop.s_result.classdef - c_class_ = hop.inputconst(ootype.Void, "class_") - v_class = hop.genop('oogetfield', [v_meta, c_class_], - resulttype=ootype.Class) resulttype = getinstancerepr(hop.rtyper, classdef).lowleveltype - v_instance = hop.genop('runtimenew', [v_class], resulttype=resulttype) - c_meta = hop.inputconst(ootype.Void, "meta") - hop.genop('oosetfield', [v_instance, c_meta, v_meta], - resulttype=ootype.Void) - return v_instance + # convert v_class from META to ootype.Class if necessary: + v_class = get_type_repr(hop.rtyper).fromclasstype(v_class, hop.llops) + return hop.genop('runtimenew', [v_class], resulttype=resulttype) + + def getlowleveltype(self): + classdescs = self.s_pbc.descriptions.keys() + # if any of the classdefs get the lowleveltype ootype.Class, + # we can only pick ootype.Class for us too. Otherwise META. + for classdesc in classdescs: + for classdef in classdesc.getallclassdefs(): + r_class = getclassrepr(self.rtyper, classdef) + if r_class.lowleveltype == ootype.Class: + return ootype.Class + else: + return META def row_method_name(methodname, rowname): Modified: pypy/dist/pypy/rpython/rclass.py ============================================================================== --- pypy/dist/pypy/rpython/rclass.py (original) +++ pypy/dist/pypy/rpython/rclass.py Thu Jul 3 15:38:13 2008 @@ -61,7 +61,8 @@ raise TyperError("not a subclass of %r: %r" % ( self.classdef.name, desc)) - return getclassrepr(self.rtyper, subclassdef).getruntime() + r_subclass = getclassrepr(self.rtyper, subclassdef) + return r_subclass.getruntime(self.lowleveltype) def convert_const(self, value): if not isinstance(value, (type, types.ClassType)): Modified: pypy/dist/pypy/rpython/rpbc.py ============================================================================== --- pypy/dist/pypy/rpython/rpbc.py (original) +++ pypy/dist/pypy/rpython/rpbc.py Thu Jul 3 15:38:13 2008 @@ -646,7 +646,7 @@ if s_pbc.is_constant(): self.lowleveltype = Void else: - self.lowleveltype = rtyper.type_system.rclass.CLASSTYPE + self.lowleveltype = self.getlowleveltype() def get_access_set(self, attrname): """Return the ClassAttrFamily corresponding to accesses to 'attrname' @@ -669,7 +669,9 @@ raise TyperError("%r not in %r" % (cls, self)) if self.lowleveltype is Void: return None - return rclass.get_type_repr(self.rtyper).convert_desc(desc) + subclassdef = desc.getuniqueclassdef() + r_subclass = rclass.getclassrepr(self.rtyper, subclassdef) + return r_subclass.getruntime(self.lowleveltype) def convert_const(self, cls): if cls is None: @@ -778,8 +780,6 @@ if r_clspbc.lowleveltype is Void: return inputconst(r_cls, r_clspbc.s_pbc.const) # convert from ptr-to-object-vtable to ptr-to-more-precise-vtable - assert (r_clspbc.lowleveltype == - r_clspbc.rtyper.type_system.rclass.CLASSTYPE) return r_cls.fromclasstype(v, llops) class __extend__(pairtype(AbstractClassesPBCRepr, AbstractClassesPBCRepr)): Modified: pypy/dist/pypy/rpython/test/test_normalizecalls.py ============================================================================== --- pypy/dist/pypy/rpython/test/test_normalizecalls.py (original) +++ pypy/dist/pypy/rpython/test/test_normalizecalls.py Thu Jul 3 15:38:13 2008 @@ -219,6 +219,7 @@ def test_add_more_subclasses(self): from pypy.rpython import rclass from pypy.rpython.lltypesystem.rclass import ll_issubclass + from pypy.rpython.lltypesystem.rclass import CLASSTYPE class Sub3(PBase): def newmethod(self): return 3 @@ -232,9 +233,9 @@ bk = translator.annotator.bookkeeper rtyper = translator.rtyper base_classdef = bk.getuniqueclassdef(PBase) - base_vtable = rclass.getclassrepr(rtyper, base_classdef).getruntime() + base_vtable = rclass.getclassrepr(rtyper, base_classdef).getruntime(CLASSTYPE) sub3_classdef = bk.getuniqueclassdef(Sub3) - sub3_vtable = rclass.getclassrepr(rtyper, sub3_classdef).getruntime() + sub3_vtable = rclass.getclassrepr(rtyper, sub3_classdef).getruntime(CLASSTYPE) assert ll_issubclass(sub3_vtable, base_vtable) assert not ll_issubclass(base_vtable, sub3_vtable) Modified: pypy/dist/pypy/rpython/test/test_rbuiltin.py ============================================================================== --- pypy/dist/pypy/rpython/test/test_rbuiltin.py (original) +++ pypy/dist/pypy/rpython/test/test_rbuiltin.py Thu Jul 3 15:38:13 2008 @@ -530,24 +530,19 @@ class TestOOtype(BaseTestRbuiltin, OORtypeMixin): - def test_instantiate_meta(self): - class A: - pass - def f(): - return instantiate(A) - res = self.interpret(f, []) - assert res.meta # check that it's not null - def test_instantiate_multiple_meta(self): class A: - pass + x = 2 class B(A): - pass + x = 3 + def do_stuff(cls): + return cls.x def f(i): if i == 1: cls = A else: cls = B + do_stuff(cls) return instantiate(cls) res = self.interpret(f, [1]) - assert res.meta # check that it's not null + assert res.getmeta() # check that it exists Modified: pypy/dist/pypy/rpython/test/test_rclass.py ============================================================================== --- pypy/dist/pypy/rpython/test/test_rclass.py (original) +++ pypy/dist/pypy/rpython/test/test_rclass.py Thu Jul 3 15:38:13 2008 @@ -97,16 +97,31 @@ assert res == 4 def test_runtime_exception(self): + class MyExc(Exception): + pass + class Sub1(MyExc): + pass + class Sub2(MyExc): + pass def pick(flag): if flag: - return TypeError + return Sub1 else: - return ValueError - def f(flag): + return Sub2 + def g(flag): ex = pick(flag) raise ex() - self.interpret_raises(TypeError, f, [True]) - self.interpret_raises(ValueError, f, [False]) + def f(flag): + try: + g(flag) + except Sub1: + return 1 + except Sub2: + return 2 + else: + return 3 + assert self.interpret(f, [True]) == 1 + assert self.interpret(f, [False]) == 2 def test_classattr_as_defaults(self): def dummyfn(): @@ -815,8 +830,8 @@ t.buildrtyper(type_system=self.type_system).specialize() graph = graphof(t, f) TYPEA = graph.startblock.operations[0].args[0].value - TYPEB = graph.startblock.operations[2].args[0].value - TYPEC = graph.startblock.operations[4].args[0].value + TYPEB = graph.startblock.operations[1].args[0].value + TYPEC = graph.startblock.operations[2].args[0].value _, destra = TYPEA._lookup("o__del__") _, destrb = TYPEB._lookup("o__del__") _, destrc = TYPEC._lookup("o__del__") Modified: pypy/dist/pypy/translator/backendopt/inline.py ============================================================================== --- pypy/dist/pypy/translator/backendopt/inline.py (original) +++ pypy/dist/pypy/translator/backendopt/inline.py Thu Jul 3 15:38:13 2008 @@ -378,7 +378,8 @@ # this rewiring does not always succeed. in the cases where it doesn't # there will be generic code inserted rclass = self.translator.rtyper.type_system.rclass - exc_match = self.translator.rtyper.getexceptiondata().fn_exception_match + excdata = self.translator.rtyper.getexceptiondata() + exc_match = excdata.fn_exception_match for link in self.entrymap[self.graph_to_inline.exceptblock]: copiedblock = self.copy_block(link.prevblock) VALUE, copiedlink = _find_exception_type(copiedblock) @@ -388,7 +389,7 @@ classdef = self.lltype_to_classdef[VALUE] rtyper = self.translator.rtyper classrepr = rclass.getclassrepr(rtyper, classdef) - vtable = classrepr.getruntime() + vtable = classrepr.getruntime(excdata.lltype_of_exception_type) var_etype = copiedlink.args[0] var_evalue = copiedlink.args[1] for exceptionlink in afterblock.exits[1:]: Modified: pypy/dist/pypy/translator/cli/database.py ============================================================================== --- pypy/dist/pypy/translator/cli/database.py (original) +++ pypy/dist/pypy/translator/cli/database.py Thu Jul 3 15:38:13 2008 @@ -7,9 +7,9 @@ from pypy.translator.cli.comparer import EqualityComparer from pypy.translator.cli.node import Node from pypy.translator.cli.support import string_literal, Counter +from pypy.translator.cli.cts import types from pypy.rpython.ootypesystem import ootype from pypy.rpython.ootypesystem.module import ll_os -from pypy.translator.cli.opcodes import opcodes from pypy.translator.cli import dotnet from pypy.rlib.objectmodel import CDefinedIntSymbolic from pypy.translator.oosupport.database import Database as OODatabase @@ -130,6 +130,8 @@ return name def class_name(self, INSTANCE): + if INSTANCE is ootype.ROOT: + return types.object.classname() try: NATIVE_INSTANCE = INSTANCE._hints['NATIVE_INSTANCE'] return NATIVE_INSTANCE._name Modified: pypy/dist/pypy/translator/cli/function.py ============================================================================== --- pypy/dist/pypy/translator/cli/function.py (original) +++ pypy/dist/pypy/translator/cli/function.py Thu Jul 3 15:38:13 2008 @@ -39,7 +39,7 @@ def record_ll_meta_exc(self, ll_meta_exc): # record the type only if it doesn't belong to a native_class - ll_exc = ll_meta_exc._inst.class_._INSTANCE + ll_exc = ll_meta_exc._INSTANCE NATIVE_INSTANCE = ll_exc._hints.get('NATIVE_INSTANCE', None) if NATIVE_INSTANCE is None: OOFunction.record_ll_meta_exc(self, ll_meta_exc) @@ -107,7 +107,7 @@ def begin_catch(self, llexitcase): ll_meta_exc = llexitcase - ll_exc = ll_meta_exc._inst.class_._INSTANCE + ll_exc = ll_meta_exc._INSTANCE cts_exc = self.cts.lltype_to_cts(ll_exc) self.ilasm.begin_catch(cts_exc.classname()) @@ -131,7 +131,7 @@ if isinstance(link.last_exception, flowmodel.Variable): self.ilasm.opcode('dup') self.store(link.last_exc_value) - self.ilasm.get_field(('class Object_meta', 'Object', 'meta')) + self.ilasm.call_method('[mscorlib]System.Type object::GetType()', virtual=True) self.store(link.last_exception) else: self.store(link.last_exc_value) Modified: pypy/dist/pypy/translator/cli/opcodes.py ============================================================================== --- pypy/dist/pypy/translator/cli/opcodes.py (original) +++ pypy/dist/pypy/translator/cli/opcodes.py Thu Jul 3 15:38:13 2008 @@ -49,6 +49,7 @@ 'cli_fieldinfo_for_const': [FieldInfoForConst], 'oois': 'ceq', 'oononnull': [PushAllArgs, 'ldnull', 'ceq']+Not, + 'classof': [PushAllArgs, 'callvirt instance [mscorlib]System.Type object::GetType()'], 'instanceof': [CastTo, 'ldnull', 'cgt.un'], 'subclassof': [PushAllArgs, 'call bool [pypylib]pypy.runtime.Utils::SubclassOf(class [mscorlib]System.Type, class[mscorlib]System.Type)'], 'ooidentityhash': [PushAllArgs, 'callvirt instance int32 object::GetHashCode()'], Modified: pypy/dist/pypy/translator/cli/test/runtest.py ============================================================================== --- pypy/dist/pypy/translator/cli/test/runtest.py (original) +++ pypy/dist/pypy/translator/cli/test/runtest.py Thu Jul 3 15:38:13 2008 @@ -99,7 +99,7 @@ if hasattr(self.db, 'exceptiontransformer'): ilasm.opcode('call', 'bool rpyexc_occured()') ilasm.opcode('brfalse', 'print_result') # no exceptions - ilasm.opcode('call', 'Object rpyexc_fetch_value()') + ilasm.opcode('call', '[mscorlib]System.Object rpyexc_fetch_value()') ilasm.call('string class [pypylib]pypy.test.Result::FormatException(object)') ilasm.call('void class [mscorlib]System.Console::WriteLine(string)') ilasm.opcode('br', 'return') @@ -282,15 +282,22 @@ def _skip_llinterpreter(self, reason, skipLL=True, skipOO=True): pass - def interpret(self, fn, args, annotation=None, backendopt=True, exctrans=False): + def _get_backendopt(self, backendopt): + if backendopt is None: + backendopt = getattr(self, 'backendopt', True) # enable it by default + return backendopt + + def interpret(self, fn, args, annotation=None, backendopt=None, exctrans=False): + backendopt = self._get_backendopt(backendopt) f = self._compile(fn, args, annotation, backendopt=backendopt, exctrans=exctrans) res = f(*args) if isinstance(res, ExceptionWrapper): raise res return res - def interpret_raises(self, exception, fn, args, backendopt=True, exctrans=False): + def interpret_raises(self, exception, fn, args, backendopt=None, exctrans=False): import exceptions # needed by eval + backendopt = self._get_backendopt(backendopt) try: self.interpret(fn, args, backendopt=backendopt, exctrans=exctrans) except ExceptionWrapper, ex: Modified: pypy/dist/pypy/translator/cli/test/test_exception.py ============================================================================== --- pypy/dist/pypy/translator/cli/test/test_exception.py (original) +++ pypy/dist/pypy/translator/cli/test/test_exception.py Thu Jul 3 15:38:13 2008 @@ -1,81 +1,22 @@ import py from pypy.translator.cli.test.runtest import CliTest -from pypy.rpython.test.test_exception import BaseTestException +from pypy.translator.oosupport.test_template.exception import BaseTestException class TestCliException(CliTest, BaseTestException): use_exception_transformer = False + backendopt = False def interpret(self, *args, **kwds): kwds['exctrans'] = self.use_exception_transformer return CliTest.interpret(self, *args, **kwds) - def test_nested_try(self): - def helper(x): - if x == 0: - raise ValueError - def dummy(): - pass - def fn(x): - try: - try: - helper(x) - finally: - dummy() - except ValueError, e: - raise - - self.interpret_raises(ValueError, fn, [0]) - - def test_exception_not_last(self): - def helper(x): - if x == 0: - raise ValueError - def fn(x): - helper(x) - try: - helper(1) - finally: - return -1 - return x - self.interpret_raises(ValueError, fn, [0]) - def test_raise_and_catch_other(self): pass def test_raise_prebuilt_and_catch_other(self): pass - def test_missing_return_block(self): - class Base: - def foo(self): - raise ValueError - - class Derived(Base): - def foo(self): - return 42 - - def fn(x): - if x: - obj = Base() - else: - obj = Derived() - return obj.foo() - assert self.interpret(fn, [0]) == 42 - - def test_missing_handler(self): - def foo(x): - if x: - raise ValueError - - def fn(x): - try: - foo(x) - except ValueError: - raise - return 42 - assert self.interpret(fn, [0], backendopt=False) == 42 - self.interpret_raises(ValueError, fn, [1], backendopt=False) - class TestCliExceptionTransformer(TestCliException): use_exception_transformer = True + backendopt = False Modified: pypy/dist/pypy/translator/jvm/database.py ============================================================================== --- pypy/dist/pypy/translator/jvm/database.py (original) +++ pypy/dist/pypy/translator/jvm/database.py Thu Jul 3 15:38:13 2008 @@ -40,11 +40,11 @@ self._constants = {} # flowmodel.Variable --> jvm.Const - # Special fields for the Object class, see _translate_Object - self._object_interf = None - self._object_impl = None - self._object_exc_impl = None - +# # Special fields for the Object class, see _translate_Object +# self._object_interf = None +# self._object_impl = None +# self._object_exc_impl = None +# # Create information about the Main class we will build: # # It will have two static fields, 'ilink' and 'pypy'. The @@ -182,67 +182,6 @@ self.pending_node(clsobj) return clsobj - def _translate_Object(self, OBJ): - """ - We handle the class 'Object' quite specially: we translate it - into an interface with two implementations. One - implementation serves as the root of most objects, and the - other as the root for all exceptions. - """ - assert self.is_Object(OBJ) - assert OBJ._superclass == ootype.ROOT - - # Have we already translated Object? - if self._object_interf: return self._object_interf - - # Create the interface and two implementations: - def gen_name(): return self._pkg(self._uniq(OBJ._name)) - internm, implnm, exc_implnm = gen_name(), gen_name(), gen_name() - self._object_interf = node.Interface(internm) - self._object_impl = node.Class(implnm, supercls=jvm.jObject) - self._object_exc_impl = node.Class(exc_implnm, supercls=jvm.jThrowable) - self._object_impl.add_interface(self._object_interf) - self._object_exc_impl.add_interface(self._object_interf) - - # Translate the fields into properties on the interface, - # and into actual fields on the implementations. - for fieldnm, (FIELDOOTY, fielddef) in OBJ._fields.iteritems(): - if FIELDOOTY is ootype.Void: continue - fieldty = self.lltype_to_cts(FIELDOOTY) - - # Currently use hacky convention of _jvm_FieldName for the name - methodnm = "_jvm_"+fieldnm - - def getter_method_obj(node): - return jvm.Method.v(node, methodnm+"_g", [], fieldty) - def putter_method_obj(node): - return jvm.Method.v(node, methodnm+"_p", [fieldty], jvm.jVoid) - - # Add get/put methods to the interface: - prop = jvm.Property( - fieldnm, - getter_method_obj(self._object_interf), - putter_method_obj(self._object_interf), - OOTYPE=FIELDOOTY) - self._object_interf.add_property(prop) - - # Generate implementations: - def generate_impl(clsobj): - clsnm = clsobj.name - fieldobj = jvm.Field(clsnm, fieldnm, fieldty, False, FIELDOOTY) - clsobj.add_field(fieldobj, fielddef) - clsobj.add_method(node.GetterFunction( - self, clsobj, getter_method_obj(clsobj), fieldobj)) - clsobj.add_method(node.PutterFunction( - self, clsobj, putter_method_obj(clsobj), fieldobj)) - generate_impl(self._object_impl) - generate_impl(self._object_exc_impl) - - # Ensure that we generate all three classes. - self.pending_node(self._object_interf) - self.pending_node(self._object_impl) - self.pending_node(self._object_exc_impl) - def _translate_superclass_of(self, OOSUB): """ Invoked to translate OOSUB's super class. Normally just invokes @@ -250,12 +189,9 @@ make all exceptions descend from Throwable. """ OOSUPER = OOSUB._superclass - if not self.is_Object(OOSUPER): - return self.pending_class(OOSUPER) - self._translate_Object(OOSUPER) # ensure this has been done if OOSUB._name == "exceptions.Exception": - return self._object_exc_impl - return self._object_impl + return jvm.jPyPyThrowable + return self.pending_class(OOSUPER) def _translate_instance(self, OOTYPE): assert isinstance(OOTYPE, ootype.Instance) @@ -564,15 +500,6 @@ TP = annotation_to_lltype(s_tp) return self.lltype_to_cts(TP) - def exception_root_object(self): - """ - Returns a JvmType representing the version of Object that - serves as the root of all exceptions. - """ - self.lltype_to_cts(rclass.OBJECT) - assert self._object_interf - return self._object_exc_impl - # _________________________________________________________________ # Uh.... # Modified: pypy/dist/pypy/translator/jvm/node.py ============================================================================== --- pypy/dist/pypy/translator/jvm/node.py (original) +++ pypy/dist/pypy/translator/jvm/node.py Thu Jul 3 15:38:13 2008 @@ -22,7 +22,8 @@ from pypy.translator.jvm.typesystem import \ JvmGeneratedClassType, jString, jStringArray, jVoid, jThrowable, jInt, \ jObject, JvmType, jStringBuilder, jPyPyInterlink, jCallbackInterfaces, \ - JvmGeneratedInterfaceType, jPyPy, jPyPyAbstractMethodException + JvmGeneratedInterfaceType, jPyPy, jPyPyAbstractMethodException, \ + jPyPyThrowable, OBJECTGETCLASS from pypy.translator.jvm.opcodes import \ opcodes from pypy.translator.jvm.option import \ @@ -173,8 +174,7 @@ gen.goto(done_printing) gen.end_try() - jexc = self.db.exception_root_object() - gen.begin_catch(jexc) + gen.begin_catch(jPyPyThrowable) gen.emit(jvm.PYPYDUMPEXCWRAPPER) # dumps to stdout gen.end_catch() @@ -366,7 +366,7 @@ def begin_catch(self, llexitcase): ll_meta_exc = llexitcase - ll_exc = ll_meta_exc._inst.class_._INSTANCE + ll_exc = ll_meta_exc._INSTANCE jtype = self.cts.lltype_to_cts(ll_exc) assert jtype.throwable # SHOULD only try to catch subtypes of Exception self.ilasm.begin_catch(jtype) @@ -384,10 +384,11 @@ else: # the exception value is on the stack, store it in the proper place if isinstance(link.last_exception, flowmodel.Variable): - self.ilasm.emit(jvm.DUP) + # if the code that follows is interested in the class + # of the exception, extract it + self.ilasm.dup_jtype(jPyPyThrowable) self.ilasm.store(link.last_exc_value) - fld = self.db.lltype_to_cts(rclass.OBJECT).lookup_field('meta') - self.ilasm.emit(fld) + self.ilasm.emit(OBJECTGETCLASS) self.ilasm.store(link.last_exception) else: self.ilasm.store(link.last_exc_value) @@ -450,8 +451,15 @@ can_branch_directly(last_op.opname) and not_in_link_args(block.exitswitch)): + self.generator.add_comment( + "short-circuit final comparison on %s, block has %d ops" % ( + block.exitswitch, len(block.operations))) + for op in block.operations[:-1]: self._render_op(op) + + self.generator.add_comment( + "inlining comparison: %r" % (last_op),) for arg in last_op.args: self.ilasm.load(arg) truelink, falselink = true_false_exits() @@ -503,12 +511,12 @@ self.ilasm.load(exc) # Check whether the static type is known to be throwable. - # If not, emit a CHECKCAST to the base exception type. + # If not, emit a CHECKCAST to throwable. # According to Samuele, no non-Exceptions should be thrown, # but this is not enforced by the RTyper or annotator. jtype = self.db.lltype_to_cts(exc.concretetype) if not jtype.throwable: - self.ilasm.downcast_jtype(self.db.exception_root_object()) + self.ilasm.downcast_jtype(jThrowable) self.ilasm.throw() Modified: pypy/dist/pypy/translator/jvm/opcodes.py ============================================================================== --- pypy/dist/pypy/translator/jvm/opcodes.py (original) +++ pypy/dist/pypy/translator/jvm/opcodes.py Thu Jul 3 15:38:13 2008 @@ -79,6 +79,7 @@ 'oodowncast': [DownCast, StoreResult], 'instanceof': [CastTo, StoreResult], 'subclassof': [PushAllArgs, jvm.SWAP, jvm.CLASSISASSIGNABLEFROM, StoreResult], + 'classof': [PushAllArgs, jvm.OBJECTGETCLASS, StoreResult], 'ooidentityhash': [PushAllArgs, jvm.OBJHASHCODE, StoreResult], 'oohash': [PushAllArgs, jvm.OBJHASHCODE, StoreResult], 'oostring': [OOString, StoreResult], Modified: pypy/dist/pypy/translator/jvm/test/test_exception.py ============================================================================== --- pypy/dist/pypy/translator/jvm/test/test_exception.py (original) +++ pypy/dist/pypy/translator/jvm/test/test_exception.py Thu Jul 3 15:38:13 2008 @@ -1,57 +1,11 @@ import py from pypy.translator.jvm.test.runtest import JvmTest -from pypy.rpython.test.test_exception import BaseTestException +from pypy.translator.oosupport.test_template.exception import BaseTestException class TestJvmException(JvmTest, BaseTestException): - def test_nested_try(self): - def helper(x): - if x == 0: - raise ValueError - def dummy(): - pass - def fn(x): - try: - try: - helper(x) - finally: - dummy() - except ValueError, e: - raise - - self.interpret_raises(ValueError, fn, [0]) - - def test_exception_not_last(self): - def helper(x): - if x == 0: - raise ValueError - def fn(x): - helper(x) - try: - helper(1) - finally: - return -1 - return x - self.interpret_raises(ValueError, fn, [0]) def test_raise_and_catch_other(self): pass def test_raise_prebuilt_and_catch_other(self): pass - - def test_missing_return_block(self): - class Base: - def foo(self): - raise ValueError - - class Derived(Base): - def foo(self): - return 42 - - def fn(x): - if x: - obj = Base() - else: - obj = Derived() - return obj.foo() - assert self.interpret(fn, [0]) == 42 Modified: pypy/dist/pypy/translator/jvm/typesystem.py ============================================================================== --- pypy/dist/pypy/translator/jvm/typesystem.py (original) +++ pypy/dist/pypy/translator/jvm/typesystem.py Thu Jul 3 15:38:13 2008 @@ -186,6 +186,7 @@ jCharClass = JvmClassType('java.lang.Character') jBoolClass = JvmClassType('java.lang.Boolean') jThrowable = JvmClassType('java.lang.Throwable', throwable=True) +jPyPyThrowable = JvmClassType('pypy.PyPyThrowable', throwable=True) jObject = JvmClassType('java.lang.Object') jString = JvmClassType('java.lang.String') jCharSequence = JvmClassType('java.lang.CharSequence') From arigo at codespeak.net Thu Jul 3 16:54:10 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 3 Jul 2008 16:54:10 +0200 (CEST) Subject: [pypy-svn] r56260 - in pypy/dist/pypy/objspace/std: . test Message-ID: <20080703145410.C31FB169F9F@codespeak.net> Author: arigo Date: Thu Jul 3 16:54:07 2008 New Revision: 56260 Modified: pypy/dist/pypy/objspace/std/model.py pypy/dist/pypy/objspace/std/register_all.py pypy/dist/pypy/objspace/std/setobject.py pypy/dist/pypy/objspace/std/slicetype.py pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py pypy/dist/pypy/objspace/std/test/test_set.py Log: Fix some failures about sets that only show up with builtinshortcut. Modified: pypy/dist/pypy/objspace/std/model.py ============================================================================== --- pypy/dist/pypy/objspace/std/model.py (original) +++ pypy/dist/pypy/objspace/std/model.py Thu Jul 3 16:54:07 2008 @@ -248,6 +248,7 @@ def get_typeorder_with_empty_usersubcls(self): if self._typeorder_with_empty_usersubcls is None: from pypy.interpreter.typedef import enum_interplevel_subclasses + from pypy.objspace.std import stdtypedef result = self.typeorder.copy() for cls in self.typeorder: if (hasattr(cls, 'typedef') and @@ -255,9 +256,14 @@ subclslist = enum_interplevel_subclasses(cls) for subcls in subclslist: if cls in subcls.__bases__: # only direct subclasses - result[subcls] = [(W_Root, None)] - # W_Root="ANY" which always matches, - # even user subclasses + # for user subclasses we only accept "generic" + # matches: "typedef.any" is the applevel-type-based + # matching, and "W_Root" is ANY. + matches = [] + if isinstance(cls.typedef, stdtypedef.StdTypeDef): + matches.append((cls.typedef.any, None)) + matches.append((W_Root, None)) + result[subcls] = matches self._typeorder_with_empty_usersubcls = result return self._typeorder_with_empty_usersubcls Modified: pypy/dist/pypy/objspace/std/register_all.py ============================================================================== --- pypy/dist/pypy/objspace/std/register_all.py (original) +++ pypy/dist/pypy/objspace/std/register_all.py Thu Jul 3 16:54:07 2008 @@ -14,6 +14,7 @@ """ from pypy.objspace.std.objspace import StdObjSpace from pypy.objspace.std.model import W_ANY, W_Object + from pypy.objspace.std.stdtypedef import StdTypeDef namespaces = list(alt_ns) + [StdObjSpace.MM, StdObjSpace] for name, obj in module_dict.items(): @@ -32,6 +33,10 @@ icls = (module_dict.get('W_%s' % i) or module_dict.get('W_%sObject' % i)) if icls is None: + x = module_dict.get(i) + if isinstance(x, StdTypeDef): + icls = x.any + if icls is None: raise ValueError, \ "no W_%s or W_%sObject for the definition of %s" % ( i, i, name) Modified: pypy/dist/pypy/objspace/std/setobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/setobject.py (original) +++ pypy/dist/pypy/objspace/std/setobject.py Thu Jul 3 16:54:07 2008 @@ -3,6 +3,8 @@ from pypy.rlib.objectmodel import r_dict from pypy.rlib.rarithmetic import intmask, r_uint from pypy.interpreter import gateway +from pypy.objspace.std.settype import set_typedef as settypedef +from pypy.objspace.std.frozensettype import frozenset_typedef as frozensettypedef class W_BaseSetObject(W_Object): @@ -119,11 +121,11 @@ else: return False -def _is_eq(w_left, w_right): - if len(w_left.setdata) != len(w_right.setdata): +def _is_eq(ld, rd): + if len(ld) != len(rd): return False - for w_key in w_left.setdata: - if w_key not in w_right.setdata: + for w_key in ld: + if w_key not in rd: return False return True @@ -191,6 +193,7 @@ #end helper functions def set_update__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _union_dict(ld, rd, True) return space.w_None @@ -231,6 +234,7 @@ return space.w_None def set_difference__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _difference_dict(ld, rd, False) return w_left._newobj(space, new_ld) @@ -252,6 +256,7 @@ def set_difference_update__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _difference_dict(ld, rd, True) return space.w_None @@ -270,29 +275,58 @@ inplace_sub__Set_Frozenset = inplace_sub__Set_Set def eq__Set_Set(space, w_left, w_other): - return space.wrap(_is_eq(w_left, w_other)) + # optimization only (the general case is eq__Set_settypedef) + return space.wrap(_is_eq(w_left.setdata, w_other.setdata)) eq__Set_Frozenset = eq__Set_Set eq__Frozenset_Frozenset = eq__Set_Set eq__Frozenset_Set = eq__Set_Set +def eq__Set_settypedef(space, w_left, w_other): + rd = make_setdata_from_w_iterable(space, w_other) + return space.wrap(_is_eq(w_left.setdata, rd)) + +eq__Set_frozensettypedef = eq__Set_settypedef +eq__Frozenset_settypedef = eq__Set_settypedef +eq__Frozenset_frozensettypedef = eq__Set_settypedef + def eq__Set_ANY(space, w_left, w_other): + # workaround to have "set() == 42" return False instead of falling + # back to cmp(set(), 42) because the latter raises a TypeError return space.w_False eq__Frozenset_ANY = eq__Set_ANY +def ne__Set_ANY(space, w_left, w_other): + # more workarounds + return space.w_True + +ne__Frozenset_ANY = ne__Set_ANY + def contains__Set_Set(space, w_left, w_other): + # optimization only (for the case __Set_settypedef) w_f = space.newfrozenset(w_other.setdata) return space.newbool(w_f in w_left.setdata) contains__Frozenset_Set = contains__Set_Set +def contains__Set_settypedef(space, w_left, w_other): + # This is the general case to handle 'set in set' or 'set in + # frozenset'. We need this in case w_other is of type 'set' but the + # case 'contains__Set_Set' is not selected by the multimethod logic, + # which can occur (see test_builtinshortcut). + w_f = space.newfrozenset(make_setdata_from_w_iterable(space, w_other)) + return space.newbool(w_f in w_left.setdata) + +contains__Frozenset_settypedef = contains__Set_settypedef + def contains__Set_ANY(space, w_left, w_other): return space.newbool(w_other in w_left.setdata) contains__Frozenset_ANY = contains__Set_ANY def set_issubset__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) if space.is_w(w_left, w_other): return space.w_True ld, rd = w_left.setdata, w_other.setdata @@ -325,9 +359,11 @@ le__Set_Set = set_issubset__Set_Set le__Set_Frozenset = set_issubset__Set_Set +le__Frozenset_Set = set_issubset__Set_Set le__Frozenset_Frozenset = set_issubset__Set_Set def set_issuperset__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) if space.is_w(w_left, w_other): return space.w_True @@ -361,18 +397,49 @@ ge__Set_Set = set_issuperset__Set_Set ge__Set_Frozenset = set_issuperset__Set_Set +ge__Frozenset_Set = set_issuperset__Set_Set ge__Frozenset_Frozenset = set_issuperset__Set_Set +# automatic registration of "lt(x, y)" as "not ge(y, x)" would not give the +# correct answer here! +def lt__Set_Set(space, w_left, w_other): + if _is_eq(w_left.setdata, w_other.setdata): + return space.w_False + else: + return le__Set_Set(space, w_left, w_other) + +lt__Set_Frozenset = lt__Set_Set +lt__Frozenset_Set = lt__Set_Set +lt__Frozenset_Frozenset = lt__Set_Set + +def gt__Set_Set(space, w_left, w_other): + if _is_eq(w_left.setdata, w_other.setdata): + return space.w_False + else: + return ge__Set_Set(space, w_left, w_other) + +gt__Set_Frozenset = gt__Set_Set +gt__Frozenset_Set = gt__Set_Set +gt__Frozenset_Frozenset = gt__Set_Set + + def set_discard__Set_Set(space, w_left, w_item): + # optimization only (the general case is set_discard__Set_settypedef) w_f = space.newfrozenset(w_item.setdata) if w_f in w_left.setdata: del w_left.setdata[w_f] +def set_discard__Set_settypedef(space, w_left, w_item): + w_f = space.newfrozenset(make_setdata_from_w_iterable(space, w_item)) + if w_f in w_left.setdata: + del w_left.setdata[w_f] + def set_discard__Set_ANY(space, w_left, w_item): if w_item in w_left.setdata: del w_left.setdata[w_item] def set_remove__Set_Set(space, w_left, w_item): + # optimization only (the general case is set_remove__Set_settypedef) w_f = space.newfrozenset(w_item.setdata) try: del w_left.setdata[w_f] @@ -380,6 +447,14 @@ raise OperationError(space.w_KeyError, space.call_method(w_item,'__repr__')) +def set_remove__Set_settypedef(space, w_left, w_item): + w_f = space.newfrozenset(make_setdata_from_w_iterable(space, w_item)) + try: + del w_left.setdata[w_f] + except KeyError: + raise OperationError(space.w_KeyError, + space.call_method(w_item,'__repr__')) + def set_remove__Set_ANY(space, w_left, w_item): try: del w_left.setdata[w_item] @@ -416,6 +491,7 @@ return w_value def set_intersection__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _intersection_dict(ld, rd, False) return w_left._newobj(space,new_ld) @@ -437,6 +513,7 @@ and__Frozenset_Frozenset = set_intersection__Set_Set def set_intersection_update__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _intersection_dict(ld, rd, True) return space.w_None @@ -455,6 +532,7 @@ inplace_and__Set_Frozenset = inplace_and__Set_Set def set_symmetric_difference__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _symmetric_difference_dict(ld, rd, False) return w_left._newobj(space, new_ld) @@ -479,6 +557,7 @@ set_symmetric_difference__Set_ANY def set_symmetric_difference_update__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _symmetric_difference_dict(ld, rd, True) return space.w_None @@ -498,6 +577,7 @@ inplace_xor__Set_Frozenset = inplace_xor__Set_Set def set_union__Set_Set(space, w_left, w_other): + # optimization only (the general case works too) ld, rd = w_left.setdata, w_other.setdata new_ld, rd = _union_dict(ld, rd, False) return w_left._newobj(space, new_ld) @@ -528,13 +608,14 @@ iter__Frozenset = iter__Set -def cmp__Set_Set(space, w_left, w_other): +def cmp__Set_settypedef(space, w_left, w_other): + # hack hack until we get the expected result raise OperationError(space.w_TypeError, space.wrap('cannot compare sets using cmp()')) -cmp__Set_Frozenset = cmp__Set_Set -cmp__Frozenset_Frozenset = cmp__Set_Set -cmp__Frozenset_Set = cmp__Set_Set +cmp__Set_frozensettypedef = cmp__Set_settypedef +cmp__Frozenset_settypedef = cmp__Set_settypedef +cmp__Frozenset_frozensettypedef = cmp__Set_settypedef def init__Set(space, w_set, __args__): w_iterable, = __args__.parse('set', @@ -551,15 +632,6 @@ hash__Frozenset(space, w_set) app = gateway.applevel(""" - def ne__Set_ANY(s, o): - return not s == o - - def gt__Set_Set(s, o): - return s != o and s.issuperset(o) - - def lt__Set_Set(s, o): - return s != o and s.issubset(o) - def repr__Set(s): return '%s(%s)' % (s.__class__.__name__, [x for x in s]) @@ -569,19 +641,6 @@ """, filename=__file__) -ne__Set_ANY = app.interphook("ne__Set_ANY") -ne__Frozenset_ANY = ne__Set_ANY - -gt__Set_Set = app.interphook("gt__Set_Set") -gt__Set_Frozenset = gt__Set_Set -gt__Frozenset_Set = gt__Set_Set -gt__Frozenset_Frozenset = gt__Set_Set - -lt__Set_Set = app.interphook("lt__Set_Set") -lt__Set_Frozenset = lt__Set_Set -lt__Frozenset_Set = lt__Set_Set -lt__Frozenset_Frozenset = lt__Set_Set - repr__Set = app.interphook('repr__Set') repr__Frozenset = app.interphook('repr__Set') Modified: pypy/dist/pypy/objspace/std/slicetype.py ============================================================================== --- pypy/dist/pypy/objspace/std/slicetype.py (original) +++ pypy/dist/pypy/objspace/std/slicetype.py Thu Jul 3 16:54:07 2008 @@ -84,4 +84,5 @@ stop = slicewprop('w_stop'), step = slicewprop('w_step'), ) +slice_typedef.acceptable_as_base_class = False slice_typedef.registermethods(globals()) Modified: pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py (original) +++ pypy/dist/pypy/objspace/std/test/test_builtinshortcut.py Thu Jul 3 16:54:07 2008 @@ -1,4 +1,5 @@ from pypy.objspace.std.test import test_userobject +from pypy.objspace.std.test import test_set WITH_BUILTINSHORTCUT = {'objspace.std.builtinshortcut': True} @@ -14,10 +15,26 @@ cls.space = conftest.gettestobjspace(**WITH_BUILTINSHORTCUT) def test_frozen_subtype(self): - skip("in-progress") class S(set): pass - assert S("abc") == set("abc") assert set("abc") == S("abc") + assert S("abc") == set("abc") class F(frozenset): pass - assert F("abc") == frozenset("abc") assert frozenset("abc") == F("abc") + assert F("abc") == frozenset("abc") + + assert S("abc") in set([frozenset("abc")]) + assert F("abc") in set([frozenset("abc")]) + + s = set([frozenset("abc")]) + s.discard(S("abc")) + assert not s + + s = set([frozenset("abc")]) + s.discard(F("abc")) + assert not s + +class AppTestSet(test_set.AppTestAppSetTest): + # this tests tons of funny comparison combinations that can easily go wrong + def setup_class(cls): + from pypy import conftest + cls.space = conftest.gettestobjspace(**WITH_BUILTINSHORTCUT) Modified: pypy/dist/pypy/objspace/std/test/test_set.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_set.py (original) +++ pypy/dist/pypy/objspace/std/test/test_set.py Thu Jul 3 16:54:07 2008 @@ -56,3 +56,18 @@ def test_compare(self): raises(TypeError, cmp, set('abc'), set('abd')) + assert set('abc') != 'abc' + raises(TypeError, "set('abc') < 42") + assert not (set('abc') < set('def')) + assert not (set('abc') <= frozenset('abd')) + assert not (set('abc') < frozenset('abd')) + assert not (set('abc') >= frozenset('abd')) + assert not (set('abc') > frozenset('abd')) + assert set('abc') <= frozenset('abc') + assert set('abc') >= frozenset('abc') + assert set('abc') <= frozenset('abcd') + assert set('abc') >= frozenset('ab') + assert set('abc') < frozenset('abcd') + assert set('abc') > frozenset('ab') + assert not (set('abc') < frozenset('abc')) + assert not (set('abc') > frozenset('abc')) From bgola at codespeak.net Thu Jul 3 20:10:48 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Thu, 3 Jul 2008 20:10:48 +0200 (CEST) Subject: [pypy-svn] r56263 - pypy/branch/2.5-features/pypy/interpreter/pyparser/data Message-ID: <20080703181048.886AC169EF8@codespeak.net> Author: bgola Date: Thu Jul 3 20:10:47 2008 New Revision: 56263 Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 Log: applying to grammar2.5 the differences between PyPy and Cpython grammar2.4 Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 Thu Jul 3 20:10:47 2008 @@ -50,8 +50,8 @@ augassign: ('+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' | '<<=' | '>>=' | '**=' | '//=') # For normal assignments, additional restrictions enforced by the interpreter -print_stmt: 'print' ( [ test (',' test)* [','] ] | - '>>' test [ (',' test)+ [','] ] ) +# print_stmt: 'print' ( [ test (',' test)* [','] ] | +print_stmt: 'print' ( '>>' test [ (',' test)+ [','] ] | [ test (',' test)* [','] ] ) del_stmt: 'del' exprlist pass_stmt: 'pass' flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt @@ -102,7 +102,8 @@ and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison comparison: expr (comp_op expr)* -comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' +# comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' +comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is' 'not'|'is' expr: xor_expr ('|' xor_expr)* xor_expr: and_expr ('^' and_expr)* and_expr: shift_expr ('&' shift_expr)* @@ -119,9 +120,11 @@ listmaker: test ( list_for | (',' test)* [','] ) testlist_gexp: test ( gen_for | (',' test)* [','] ) lambdef: 'lambda' [varargslist] ':' test -trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME +# trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME +trailer: '(' ')' | '(' arglist ')' | '[' subscriptlist ']' | '.' NAME subscriptlist: subscript (',' subscript)* [','] -subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] +# subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] +subscript: '.' '.' '.' | [test] ':' [test] [sliceop] | test sliceop: ':' [test] exprlist: expr (',' expr)* [','] testlist: test (',' test)* [','] @@ -129,7 +132,8 @@ classdef: 'class' NAME ['(' [testlist] ')'] ':' suite -arglist: (argument ',')* (argument [',']| '*' test [',' '**' test] | '**' test) +# arglist: (argument ',')* (argument [',']| '*' test [',' '**' test] | '**' test) +arglist: (argument ',')* ( '*' test [',' '**' test] | '**' test | argument | [argument ','] ) argument: test [gen_for] | test '=' test # Really [keyword '='] test list_iter: list_for | list_if From fijal at codespeak.net Thu Jul 3 21:06:01 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Thu, 3 Jul 2008 21:06:01 +0200 (CEST) Subject: [pypy-svn] r56264 - pypy/extradoc/talk/ep2008 Message-ID: <20080703190601.7889C2A8002@codespeak.net> Author: fijal Date: Thu Jul 3 21:05:58 2008 New Revision: 56264 Modified: pypy/extradoc/talk/ep2008/status.txt Log: a bit of update Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Thu Jul 3 21:05:58 2008 @@ -63,7 +63,6 @@ * Part of cpython stdlib * We use Gerhard Haering's ctypes version - (XXX link) * Works reasonably well after fixes @@ -121,14 +120,15 @@ Examples of obscure features ============================ -XXX - non-string keys in typedict +* non-string keys in typedict -XXX - list comprehension variable in __all__ +* list comprehension variable in __all__ -XXX - relying on untested and undocumented private stuff - (zipimport._zip_directory_cache) +* relying on untested and undocumented private stuff + (zipimport._zip_directory_cache) -XXX - exact strings in exceptions +* exact message matching in exception catching + code Speed - comparison with CPython =============================== @@ -147,7 +147,7 @@ * Better handling of obscure cases -* Troubles with ie communication with C +* Troubles with for example communication with C Speed - JIT generator ===================== @@ -169,6 +169,19 @@ * Both backends are progressing - very slowly though +Sandboxing +========== + +* Fully sandboxed python interpreter + +* All external calls to C goes via another + python process + +* Special library for making custom + policies + +XXX demo + Plans ===== @@ -176,7 +189,7 @@ * Support full cpython's stdlib -* Get more funding money +* Unable to segfault No-plans ======== From cfbolz at codespeak.net Thu Jul 3 21:34:51 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 3 Jul 2008 21:34:51 +0200 (CEST) Subject: [pypy-svn] r56266 - pypy/extradoc/talk/ep2008 Message-ID: <20080703193451.67F702D80B8@codespeak.net> Author: cfbolz Date: Thu Jul 3 21:34:50 2008 New Revision: 56266 Modified: pypy/extradoc/talk/ep2008/status.txt Log: Lots of tiny details. Good talk apart from that. Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Thu Jul 3 21:34:50 2008 @@ -9,46 +9,50 @@ What this talk is about ======================= -* more details recent developements +* more details about recent developments -* our plans +* our plans for the near future -* how we're going to achieve this +* how we're going to achieve them -* more or less recap of our blog +* readers of our blog might know many of those things Production ready ===================== -* We concentrated on running - existing applications on top of pypy +* We worked a lot on running + existing applications on top of PyPy * Sometimes requiring to change applications slightly * Especially refcounting details tend to be a problem open('xxx', 'w').write('stuff') -Ctypes +CTypes ====== * Official way to have bindings to - external (C) libraries for pypy + external (C) libraries for PyPy * Slow, but getting better -* Sponsored by google +* Sponsored by Google -* Can handle ie pysqlite-ctypes, pyglet, pymunk or Sole Scion +* Can handle i.e. pysqlite-ctypes, pyglet, pymunk or Sole Scion XXX demo of sole scion and pyglet -* ctypes is getting better as a side effect +* ctypes is getting better as a side effect: -Ctypes configure + * bugs found + * helper libraries + * possibility of porting to Jython/IronPython + +CTypes configure ================ * Our own small addition to general - ctypes usefulness + CTypes usefulness * Invokes C compiler for small details @@ -60,20 +64,20 @@ Sqlite ====== -* Part of cpython stdlib +* Part of cpython stdlib since 2.5 -* We use Gerhard Haering's ctypes version +* We use Gerhard Haering's CTypes version -* Works reasonably well after fixes +* Works reasonably well after some fixes Django ====== -* We run (almost) unmodified django +* We run (almost) unmodified Django * Only sqlite DB backend for now -* We'll run 1.0 definitely +* Cooperation with Django people to make sure that Django 1.0 works with PyPy XXX demo @@ -83,7 +87,7 @@ * Worked almost out of the box once eggs are working (1 day) -* No sqlalchemy yet, obscure problems +* No SQLAlchemy yet, obscure problems ahead XXX demo @@ -95,15 +99,15 @@ * Nevow works -* We don't support pycrypto nor pyopenssl and we - won't +* We don't support PyCrypto nor PyOpenSSL and we + won't anytime soon (if nobody contributes CTypes versions) Other software ============== -* bittorrent +* BitTorrent -* pypy translation toolchain +* PyPy translation toolchain * various smaller things, templating engines, most pure-python software @@ -115,12 +119,12 @@ not to rely on it * Generally it's fairly unlikely we'll ever try - to support stuff without tests + to support stuff without tests XXX what does this mean? we already support things that are untested in CPython Examples of obscure features ============================ -* non-string keys in typedict +* non-string keys in __dict__ of types * list comprehension variable in __all__ @@ -134,13 +138,13 @@ =============================== * We're something between 0.8-2x slower than - cpython on various benchmarks. + CPython on various benchmarks. -* gcbench - 0.8 (fast gc) +* gcbench - 0.8 (because of our faster GC) * We're progressing slowly -Speed - decent gcs +Speed - decent GCs ================== * Faster than refcounting @@ -161,7 +165,7 @@ Other backends ============== -* pypy-jvm runs! +* PyPy-jvm runs! * More integration between pypy-cli and .NET @@ -169,6 +173,8 @@ * Both backends are progressing - very slowly though +* contributors wanted! + Sandboxing ========== @@ -187,9 +193,9 @@ * More JIT - faster Python -* Support full cpython's stdlib +* Support full CPython's stdlib -* Unable to segfault +* make PyPy impossible to segfault - close to that already No-plans ======== From fijal at codespeak.net Thu Jul 3 21:42:30 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Thu, 3 Jul 2008 21:42:30 +0200 (CEST) Subject: [pypy-svn] r56267 - pypy/extradoc/talk/ep2008 Message-ID: <20080703194230.04F2D2D80CC@codespeak.net> Author: fijal Date: Thu Jul 3 21:42:29 2008 New Revision: 56267 Modified: pypy/extradoc/talk/ep2008/status.txt Log: Add a slide about cleanup sprint Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Thu Jul 3 21:42:29 2008 @@ -188,6 +188,16 @@ XXX demo +Cleanup sprint +============== + +* Got rid of semi-cool semi-working features + +* Reduced code size + +* Allowed us to progress forward into + advanced features + Plans ===== From bgola at codespeak.net Thu Jul 3 21:51:45 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Thu, 3 Jul 2008 21:51:45 +0200 (CEST) Subject: [pypy-svn] r56268 - pypy/branch/2.5-features/pypy/interpreter/pyparser/data Message-ID: <20080703195145.2F9EE16855F@codespeak.net> Author: bgola Date: Thu Jul 3 21:51:43 2008 New Revision: 56268 Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 Log: another change to grammar (also in 2.4) Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 Thu Jul 3 21:51:43 2008 @@ -134,7 +134,8 @@ # arglist: (argument ',')* (argument [',']| '*' test [',' '**' test] | '**' test) arglist: (argument ',')* ( '*' test [',' '**' test] | '**' test | argument | [argument ','] ) -argument: test [gen_for] | test '=' test # Really [keyword '='] test +#argument: test [gen_for] | test '=' test # Really [keyword '='] test +argument: [test '='] test [gen_for] # Really [keyword '='] test list_iter: list_for | list_if list_for: 'for' exprlist 'in' testlist_safe [list_iter] From cfbolz at codespeak.net Thu Jul 3 21:55:25 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 3 Jul 2008 21:55:25 +0200 (CEST) Subject: [pypy-svn] r56269 - pypy/extradoc/talk/ep2008 Message-ID: <20080703195525.5FF96698079@codespeak.net> Author: cfbolz Date: Thu Jul 3 21:55:24 2008 New Revision: 56269 Modified: pypy/extradoc/talk/ep2008/status.txt Log: like this, maybe? Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Thu Jul 3 21:55:24 2008 @@ -118,8 +118,7 @@ * There is no feature obscure enough for people not to rely on it -* Generally it's fairly unlikely we'll ever try - to support stuff without tests XXX what does this mean? we already support things that are untested in CPython +* It's extremely hard to try to run stuff on PyPy that doesn't have unittests Examples of obscure features ============================ From hpk at codespeak.net Thu Jul 3 22:30:52 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Thu, 3 Jul 2008 22:30:52 +0200 (CEST) Subject: [pypy-svn] r56275 - pypy/extradoc/talk/ep2008 Message-ID: <20080703203052.A66CB169F57@codespeak.net> Author: hpk Date: Thu Jul 3 22:30:52 2008 New Revision: 56275 Modified: pypy/extradoc/talk/ep2008/status.txt Log: review, cutting out some negative not too telling statements and adding a bit to plans from my perspective Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Thu Jul 3 22:30:52 2008 @@ -112,16 +112,8 @@ * various smaller things, templating engines, most pure-python software -Conclusion -========== - -* There is no feature obscure enough for people - not to rely on it - -* It's extremely hard to try to run stuff on PyPy that doesn't have unittests - -Examples of obscure features -============================ +Obscure differences to CPython +============================== * non-string keys in __dict__ of types @@ -133,6 +125,17 @@ * exact message matching in exception catching code +Conclusion on Compatibility +============================ + +* lessons learned: There is no feature obscure enough for people + not to rely on it. But PyPy can usually mimick CPython sufficiently. + +* pypy-c probably most compatible to CPython Interpreter + +* main blocker for running apps will be missing external modules + + Speed - comparison with CPython =============================== @@ -141,7 +144,7 @@ * gcbench - 0.8 (because of our faster GC) -* We're progressing slowly +* steady but slow progress Speed - decent GCs ================== @@ -200,16 +203,14 @@ Plans ===== -* More JIT - faster Python +* More JIT - faster Python * Support full CPython's stdlib * make PyPy impossible to segfault - close to that already -No-plans -======== +* adapt PyPy to more platforms -* RPython extensions for CPython +* aim for special optimisations, targets and security goals + and get funding for it -* Maintain arbitrary number of third-party - extension modules From jacob at codespeak.net Fri Jul 4 00:33:52 2008 From: jacob at codespeak.net (jacob at codespeak.net) Date: Fri, 4 Jul 2008 00:33:52 +0200 (CEST) Subject: [pypy-svn] r56280 - pypy/extradoc/talk/ep2008 Message-ID: <20080703223352.093B52A8093@codespeak.net> Author: jacob Date: Fri Jul 4 00:33:52 2008 New Revision: 56280 Added: pypy/extradoc/talk/ep2008/Pypy_for_the_rest_of_us.odp (contents, props changed) Log: My talk. Added: pypy/extradoc/talk/ep2008/Pypy_for_the_rest_of_us.odp ============================================================================== Binary file. No diff available. From bgola at codespeak.net Fri Jul 4 01:03:09 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Fri, 4 Jul 2008 01:03:09 +0200 (CEST) Subject: [pypy-svn] r56281 - in pypy/branch/2.5-features/pypy: interpreter lib Message-ID: <20080703230309.EE06B2A8093@codespeak.net> Author: bgola Date: Fri Jul 4 01:03:08 2008 New Revision: 56281 Modified: pypy/branch/2.5-features/pypy/interpreter/error.py pypy/branch/2.5-features/pypy/lib/_exceptions.py Log: PEP 352, BaseException... Modified: pypy/branch/2.5-features/pypy/interpreter/error.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/error.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/error.py Fri Jul 4 01:03:08 2008 @@ -175,8 +175,8 @@ elif space.full_exceptions and space.is_w(space.type(w_type), space.w_str): - # XXX warn -- deprecated - pass + space.warn("raising a string exception is deprecated", + space.w_DeprecationWarning) else: # raise X: we assume that X is an already-built instance Modified: pypy/branch/2.5-features/pypy/lib/_exceptions.py ============================================================================== --- pypy/branch/2.5-features/pypy/lib/_exceptions.py (original) +++ pypy/branch/2.5-features/pypy/lib/_exceptions.py Fri Jul 4 01:03:08 2008 @@ -19,77 +19,62 @@ recommended that user defined class based exceptions be derived from the `Exception' class, although this is currently not enforced. -Exception - | +BaseException +-- SystemExit - +-- StopIteration - +-- StandardError - | | - | +-- KeyboardInterrupt - | +-- ImportError - | +-- EnvironmentError - | | | - | | +-- IOError - | | +-- OSError - | | | - | | +-- WindowsError - | | +-- VMSError - | | - | +-- EOFError - | +-- RuntimeError - | | | - | | +-- NotImplementedError - | | - | +-- NameError - | | | - | | +-- UnboundLocalError - | | - | +-- AttributeError - | +-- SyntaxError - | | | - | | +-- IndentationError - | | | - | | +-- TabError - | | - | +-- TypeError - | +-- AssertionError - | +-- LookupError - | | | - | | +-- IndexError - | | +-- KeyError - | | - | +-- ArithmeticError - | | | - | | +-- OverflowError - | | +-- ZeroDivisionError - | | +-- FloatingPointError - | | - | +-- ValueError - | | | - | | +-- UnicodeError - | | | - | | +-- UnicodeEncodeError - | | +-- UnicodeDecodeError - | | +-- UnicodeTranslateError - | | - | +-- ReferenceError - | +-- SystemError - | +-- MemoryError - | - +---Warning - | - +-- UserWarning - +-- DeprecationWarning - +-- PendingDeprecationWarning - +-- SyntaxWarning - +-- OverflowWarning - +-- RuntimeWarning - +-- FutureWarning - +-- UnicodeWarning - +-- ImportWarning""" - -class Exception: - """Common base class for all exceptions.""" + +-- KeyboardInterrupt + +-- Exception + +-- GeneratorExit + +-- StopIteration + +-- StandardError + | +-- ArithmeticError + | | +-- FloatingPointError + | | +-- OverflowError + | | +-- ZeroDivisionError + | +-- AssertionError + | +-- AttributeError + | +-- EnvironmentError + | | +-- IOError + | | +-- OSError + | | +-- WindowsError (Windows) + | | +-- VMSError (VMS) + | +-- EOFError + | +-- ImportError + | +-- LookupError + | | +-- IndexError + | | +-- KeyError + | +-- MemoryError + | +-- NameError + | | +-- UnboundLocalError + | +-- ReferenceError + | +-- RuntimeError + | | +-- NotImplementedError + | +-- SyntaxError + | | +-- IndentationError + | | +-- TabError + | +-- SystemError + | +-- TypeError + | +-- ValueError + | | +-- UnicodeError + | | +-- UnicodeDecodeError + | | +-- UnicodeEncodeError + | | +-- UnicodeTranslateError + +-- Warning + +-- DeprecationWarning + +-- PendingDeprecationWarning + +-- RuntimeWarning + +-- SyntaxWarning + +-- UserWarning + +-- FutureWarning + +-- ImportWarning + +-- UnicodeWarning +""" + +class BaseException(object): + """Superclass representing the base of the exception hierarchy. + + The __getitem__ method is provided for backwards-compatibility + and will be deprecated at some point. + """ def __getitem__(self, idx): return self.args[idx] @@ -107,6 +92,17 @@ else: return str(args) + def __repr__(self): + if self.args: + func_args = repr(self.args) + else: + func_args = "()" + return self.__class__.__name__ + func_args + + +class Exception(BaseException): + """Common base class for all non-exit exceptions.""" + class StandardError(Exception): """Base class for all standard Python exceptions.""" @@ -293,7 +289,7 @@ class FutureWarning(Warning): """Base class for warnings about constructs that will change semantically in the future.""" -class SystemExit(Exception): +class SystemExit(BaseException): """Request to exit from the interpreter.""" def __init__(self, *args): @@ -376,7 +372,7 @@ class RuntimeWarning(Warning): """Base class for warnings about dubious runtime behavior.""" -class KeyboardInterrupt(StandardError): +class KeyboardInterrupt(BaseException): """Program interrupted by user.""" class UserWarning(Warning): From arigo at codespeak.net Fri Jul 4 10:14:12 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 4 Jul 2008 10:14:12 +0200 (CEST) Subject: [pypy-svn] r56287 - pypy/dist/pypy/translator/backendopt Message-ID: <20080704081412.1E2E4169FC2@codespeak.net> Author: arigo Date: Fri Jul 4 10:14:11 2008 New Revision: 56287 Modified: pypy/dist/pypy/translator/backendopt/removeassert.py Log: Fix this code to use a more standard interface. The previous code failed after the merging of the less-meta-instances branch. Modified: pypy/dist/pypy/translator/backendopt/removeassert.py ============================================================================== --- pypy/dist/pypy/translator/backendopt/removeassert.py (original) +++ pypy/dist/pypy/translator/backendopt/removeassert.py Fri Jul 4 10:14:11 2008 @@ -9,9 +9,9 @@ def remove_asserts(translator, graphs): rtyper = translator.rtyper + excdata = rtyper.exceptiondata clsdef = translator.annotator.bookkeeper.getuniqueclassdef(AssertionError) - r_AssertionError = rclass.getclassrepr(rtyper, clsdef) - ll_AssertionError = r_AssertionError.convert_const(AssertionError) + ll_AssertionError = excdata.get_standard_ll_exc_instance(rtyper, clsdef) total_count = [0, 0] for graph in graphs: @@ -23,8 +23,8 @@ join_blocks(graph) for link in graph.iterlinks(): if (link.target is graph.exceptblock - and isinstance(link.args[0], Constant) - and link.args[0].value == ll_AssertionError): + and isinstance(link.args[1], Constant) + and link.args[1].value == ll_AssertionError): if kill_assertion_link(graph, link): count += 1 morework = True From arigo at codespeak.net Fri Jul 4 10:20:38 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 4 Jul 2008 10:20:38 +0200 (CEST) Subject: [pypy-svn] r56288 - pypy/dist/pypy/lib/app_test Message-ID: <20080704082038.0A5F5169FC5@codespeak.net> Author: arigo Date: Fri Jul 4 10:20:37 2008 New Revision: 56288 Modified: pypy/dist/pypy/lib/app_test/test_pyexpat.py Log: Fix test, which wasn't testing anything from pypy. Modified: pypy/dist/pypy/lib/app_test/test_pyexpat.py ============================================================================== --- pypy/dist/pypy/lib/app_test/test_pyexpat.py (original) +++ pypy/dist/pypy/lib/app_test/test_pyexpat.py Fri Jul 4 10:20:37 2008 @@ -4,7 +4,7 @@ import StringIO, sys import unittest, py -import pyexpat +from pypy.lib import pyexpat #from xml.parsers import expat expat = pyexpat From arigo at codespeak.net Fri Jul 4 10:47:53 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 4 Jul 2008 10:47:53 +0200 (CEST) Subject: [pypy-svn] r56291 - pypy/dist/pypy/module/_lsprof Message-ID: <20080704084753.5B0EB16A07D@codespeak.net> Author: arigo Date: Fri Jul 4 10:47:52 2008 New Revision: 56291 Modified: pypy/dist/pypy/module/_lsprof/interp_lsprof.py Log: This __del__ was a bad idea: it's pointless because the executioncontext holds a ref to space.wrap(self) as long as the profiler is enabled; moreover it's dangerous because it might accidentally un-install another profiler that would happen to be running at that time. Modified: pypy/dist/pypy/module/_lsprof/interp_lsprof.py ============================================================================== --- pypy/dist/pypy/module/_lsprof/interp_lsprof.py (original) +++ pypy/dist/pypy/module/_lsprof/interp_lsprof.py Fri Jul 4 10:47:52 2008 @@ -223,11 +223,6 @@ space.getexecutioncontext().setllprofile(None, None) disable.unwrap_spec = ['self', ObjSpace] - def delete(self, space): - self._flush_unmatched() - space.getexecutioncontext().setllprofile(None, None) - delete.unwrap_spec = ['self', ObjSpace] - def getstats(self, space): if self.w_callable is None: factor = 1. # we measure time.time in floats @@ -251,5 +246,4 @@ enable = interp2app(W_Profiler.enable), disable = interp2app(W_Profiler.disable), getstats = interp2app(W_Profiler.getstats), - __del__ = interp2app(W_Profiler.delete), ) From arigo at codespeak.net Fri Jul 4 11:05:19 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 4 Jul 2008 11:05:19 +0200 (CEST) Subject: [pypy-svn] r56292 - pypy/dist/pypy/module/_lsprof/test Message-ID: <20080704090519.C2F8D16A07B@codespeak.net> Author: arigo Date: Fri Jul 4 11:05:19 2008 New Revision: 56292 Modified: pypy/dist/pypy/module/_lsprof/test/test_cprofile.py Log: * add a direct test. * skip the end of the failing output test. Modified: pypy/dist/pypy/module/_lsprof/test/test_cprofile.py ============================================================================== --- pypy/dist/pypy/module/_lsprof/test/test_cprofile.py (original) +++ pypy/dist/pypy/module/_lsprof/test/test_cprofile.py Fri Jul 4 11:05:19 2008 @@ -8,7 +8,82 @@ cls.w_expected_output = space.wrap(expected_output) cls.space = space cls.w_file = space.wrap(__file__) - + + def test_direct(self): + import _lsprof + def getticks(): + return len(ticks) + prof = _lsprof.Profiler(getticks, 0.25, True, False) + ticks = [] + def bar(m): + ticks.append(1) + if m == 1: + foo(42) + ticks.append(1) + def spam(m): + bar(m) + def foo(n): + bar(n) + ticks.append(1) + bar(n+1) + ticks.append(1) + spam(n+2) + prof.enable() + foo(0) + prof.disable() + assert len(ticks) == 16 + stats = prof.getstats() + entries = {} + for entry in stats: + assert hasattr(entry.code, 'co_name') + entries[entry.code.co_name] = entry + efoo = entries['foo'] + assert efoo.callcount == 2 + assert efoo.reccallcount == 1 + assert efoo.inlinetime == 1.0 + assert efoo.totaltime == 4.0 + assert len(efoo.calls) == 2 + ebar = entries['bar'] + assert ebar.callcount == 6 + assert ebar.reccallcount == 3 + assert ebar.inlinetime == 3.0 + assert ebar.totaltime == 3.5 + assert len(ebar.calls) == 1 + espam = entries['spam'] + assert espam.callcount == 2 + assert espam.reccallcount == 0 + assert espam.inlinetime == 0.0 + assert espam.totaltime == 1.0 + assert len(espam.calls) == 1 + + foo2bar, foo2spam = efoo.calls + if foo2bar.code.co_name == 'spam': + foo2bar, foo2spam = foo2spam, foo2bar + assert foo2bar.code.co_name == 'bar' + assert foo2bar.callcount == 4 + assert foo2bar.reccallcount == 2 + assert foo2bar.inlinetime == 2.0 + assert foo2bar.totaltime == 3.0 + assert foo2spam.code.co_name == 'spam' + assert foo2spam.callcount == 2 + assert foo2spam.reccallcount == 0 + assert foo2spam.inlinetime == 0.0 + assert foo2spam.totaltime == 1.0 + + bar2foo, = ebar.calls + assert bar2foo.code.co_name == 'foo' + assert bar2foo.callcount == 1 + assert bar2foo.reccallcount == 0 + assert bar2foo.inlinetime == 0.5 + assert bar2foo.totaltime == 2.0 + + spam2bar, = espam.calls + assert spam2bar.code.co_name == 'bar' + assert spam2bar.callcount == 2 + assert spam2bar.reccallcount == 0 + assert spam2bar.inlinetime == 1.0 + assert spam2bar.totaltime == 1.0 + def test_cprofile(self): import sys, os # XXX this is evil trickery to walk around the fact that we don't @@ -38,6 +113,8 @@ res, prof = do_profiling(Profile) assert res[0] == 1000 + skip("when we reach this point, things seem to work; " + "but we need to review carefully what we expect as output") for i, method in enumerate(methodnames): if not res[i + 1] == self.expected_output[method]: print method From bgola at codespeak.net Fri Jul 4 15:32:29 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Fri, 4 Jul 2008 15:32:29 +0200 (CEST) Subject: [pypy-svn] r56300 - pypy/branch/2.5-features/pypy/module/__builtin__/test Message-ID: <20080704133229.7FBA839800A@codespeak.net> Author: bgola Date: Fri Jul 4 15:32:28 2008 New Revision: 56300 Modified: pypy/branch/2.5-features/pypy/module/__builtin__/test/test_import.py Log: Fixing import warning test (copy-paste error) Modified: pypy/branch/2.5-features/pypy/module/__builtin__/test/test_import.py ============================================================================== --- pypy/branch/2.5-features/pypy/module/__builtin__/test/test_import.py (original) +++ pypy/branch/2.5-features/pypy/module/__builtin__/test/test_import.py Fri Jul 4 15:32:28 2008 @@ -122,7 +122,7 @@ try: raises(ImportWarning, imp) finally: - warnings.simplefilter('default', RuntimeWarning) + warnings.simplefilter('default', ImportWarning) def test_import_sys(self): import sys From bgola at codespeak.net Fri Jul 4 15:41:05 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Fri, 4 Jul 2008 15:41:05 +0200 (CEST) Subject: [pypy-svn] r56301 - in pypy/branch/2.5-features/pypy: interpreter/test lib/app_test Message-ID: <20080704134105.4AF2639800A@codespeak.net> Author: bgola Date: Fri Jul 4 15:41:04 2008 New Revision: 56301 Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_raise.py pypy/branch/2.5-features/pypy/lib/app_test/test_exception_extra.py Log: tests for PEP 352 (BaseException and raise 'string') Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_raise.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/test/test_raise.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/test/test_raise.py Fri Jul 4 15:41:04 2008 @@ -1,6 +1,17 @@ class AppTestRaise: + def test_arg_as_string(self): + def f(): + raise "test" + + import warnings + warnings.simplefilter('error', DeprecationWarning) + try: + raises(DeprecationWarning, f) + finally: + warnings.simplefilter('default', DeprecationWarning) + def test_control_flow(self): try: raise Exception Modified: pypy/branch/2.5-features/pypy/lib/app_test/test_exception_extra.py ============================================================================== --- pypy/branch/2.5-features/pypy/lib/app_test/test_exception_extra.py (original) +++ pypy/branch/2.5-features/pypy/lib/app_test/test_exception_extra.py Fri Jul 4 15:41:04 2008 @@ -11,3 +11,12 @@ def app_test_import(): import exceptions assert exceptions.SyntaxError is SyntaxError + +def app_test_baseexception(): + assert issubclass(Exception, BaseException) + +def app_test_systemexit(): + assert issubclass(SystemExit, BaseException) + +def app_test_keyboardinterrupt(): + assert issubclass(KeyboardInterrupt, BaseException) From bgola at codespeak.net Fri Jul 4 15:43:04 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Fri, 4 Jul 2008 15:43:04 +0200 (CEST) Subject: [pypy-svn] r56302 - in pypy/branch/2.5-features/pypy/interpreter: astcompiler test Message-ID: <20080704134304.75058398005@codespeak.net> Author: bgola Date: Fri Jul 4 15:43:03 2008 New Revision: 56302 Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/symbols.py pypy/branch/2.5-features/pypy/interpreter/test/test_syntax.py Log: removing the SyntaxWarning for conditional expressions, i think it's fully supported now Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/symbols.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/astcompiler/symbols.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/astcompiler/symbols.py Fri Jul 4 15:43:03 2008 @@ -572,8 +572,6 @@ node.value.accept(self) def visitCondExpr(self, node): - issue_warning(self.space, "conditional expression", - node.filename, node.lineno) ast.ASTVisitor.visitCondExpr(self, node) def sort(l): Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_syntax.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/test/test_syntax.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/test/test_syntax.py Fri Jul 4 15:43:03 2008 @@ -259,7 +259,7 @@ class Py25AppTest: def setup_class(self): - self.space = gettestobjspace(pyversion='2.5a') + self.space = gettestobjspace(pyversion='2.5') return class AppTestCondExpr(Py25AppTest): @@ -269,6 +269,14 @@ exec s assert x == expected + def test_condexpr_no_warning(self): + import warnings + + warnings.simplefilter('error', SyntaxWarning) + 1 if True else 2 + warnings.simplefilter('default', SyntaxWarning) + + class AppTestWith(Py25AppTest): def test_with_simple(self): From jacob at codespeak.net Fri Jul 4 17:03:25 2008 From: jacob at codespeak.net (jacob at codespeak.net) Date: Fri, 4 Jul 2008 17:03:25 +0200 (CEST) Subject: [pypy-svn] r56305 - pypy/extradoc/talk/ep2008 Message-ID: <20080704150325.7E010698079@codespeak.net> Author: jacob Date: Fri Jul 4 17:03:23 2008 New Revision: 56305 Modified: pypy/extradoc/talk/ep2008/Pypy_for_the_rest_of_us.odp Log: Modifications as per Maciek's suggestions. Modified: pypy/extradoc/talk/ep2008/Pypy_for_the_rest_of_us.odp ============================================================================== Binary files. No diff available. From fijal at codespeak.net Fri Jul 4 17:24:54 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Fri, 4 Jul 2008 17:24:54 +0200 (CEST) Subject: [pypy-svn] r56306 - pypy/extradoc/talk/ep2008 Message-ID: <20080704152454.1B925698039@codespeak.net> Author: fijal Date: Fri Jul 4 17:24:53 2008 New Revision: 56306 Modified: pypy/extradoc/talk/ep2008/status.txt Log: Simplify a bit and update a bit Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Fri Jul 4 17:24:53 2008 @@ -188,12 +188,15 @@ * Special library for making custom policies +XXX image + XXX demo Cleanup sprint ============== -* Got rid of semi-cool semi-working features +* Got rid of semi-cool semi-working + proofs of concept * Reduced code size @@ -207,10 +210,14 @@ * Support full CPython's stdlib -* make PyPy impossible to segfault - close to that already +* Aim for more funding in pushing pypy + forward (more at the next talk) + +Links +===== -* adapt PyPy to more platforms +* http://codespeak.net/pypy -* aim for special optimisations, targets and security goals - and get funding for it +* http://morepypy.blogspot.com +* this talk is already online From bgola at codespeak.net Fri Jul 4 20:26:40 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Fri, 4 Jul 2008 20:26:40 +0200 (CEST) Subject: [pypy-svn] r56309 - in pypy/branch/2.5-features/pypy/module/_file: . test Message-ID: <20080704182640.BBB102A8083@codespeak.net> Author: bgola Date: Fri Jul 4 20:26:39 2008 New Revision: 56309 Modified: pypy/branch/2.5-features/pypy/module/_file/interp_file.py pypy/branch/2.5-features/pypy/module/_file/test/test_file.py Log: __enter__ and __exit__ methods for file object Modified: pypy/branch/2.5-features/pypy/module/_file/interp_file.py ============================================================================== --- pypy/branch/2.5-features/pypy/module/_file/interp_file.py (original) +++ pypy/branch/2.5-features/pypy/module/_file/interp_file.py Fri Jul 4 20:26:39 2008 @@ -75,6 +75,14 @@ fd = stream.try_to_find_file_descriptor() self.fdopenstream(stream, fd, mode, name) + def direct___enter__(self): + return self + + def direct___exit__(self, excinfo): + self.direct_close() + # can't return close() value + return None + def direct_fdopen(self, fd, mode='r', buffering=-1): self.direct_close() self.check_mode_ok(mode) @@ -263,6 +271,11 @@ _decl(locals(), "__init__", ['self', str, str, int], """Opens a file.""") + _decl(locals(), "__enter__", ['self'], """enter__() -> self.""") + + _decl(locals(), "__exit__", ['self', Arguments], + """exit__(*excinfo) -> None. Closes the file.""") + _decl(locals(), "close", ['self'], """close() -> None or (perhaps) an integer. Close the file. Modified: pypy/branch/2.5-features/pypy/module/_file/test/test_file.py ============================================================================== --- pypy/branch/2.5-features/pypy/module/_file/test/test_file.py (original) +++ pypy/branch/2.5-features/pypy/module/_file/test/test_file.py Fri Jul 4 20:26:39 2008 @@ -222,6 +222,40 @@ print 'Passed.' +class AppTestFile25: + def setup_class(cls): + cls.space = gettestobjspace(usemodules=("_file", ), pyversion="2.5") + cls.w_temppath = cls.space.wrap( + str(py.test.ensuretemp("fileimpl").join("foo.txt"))) + cls.w_file = getfile(cls.space) + + def test___enter__(self): + f = self.file(self.temppath, 'w') + assert f.__enter__() is f + + def test___exit__(self): + f = self.file(self.temppath, 'w') + assert f.__exit__() is None + assert f.closed + + def test_file_and_with_statement(self): + s1 = """from __future__ import with_statement +with self.file(self.temppath, 'w') as f: + f.write('foo') +""" + exec s1 + assert f.closed + + s2 = """from __future__ import with_statement +with self.file(self.temppath, 'r') as f: + s = f.readline() +""" + + exec s2 + assert s == "foo" + assert f.closed + + def test_flush_at_exit(): from pypy import conftest from pypy.tool.option import make_config, make_objspace From cfbolz at codespeak.net Fri Jul 4 20:37:24 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Fri, 4 Jul 2008 20:37:24 +0200 (CEST) Subject: [pypy-svn] r56310 - pypy/branch/2.5-features/pypy/module/_file Message-ID: <20080704183724.648AF49800E@codespeak.net> Author: cfbolz Date: Fri Jul 4 20:37:22 2008 New Revision: 56310 Modified: pypy/branch/2.5-features/pypy/module/_file/interp_file.py Log: Typos in the doc strings. Arguments in the unwrap spec are usually called __args__ in the parameter names. Modified: pypy/branch/2.5-features/pypy/module/_file/interp_file.py ============================================================================== --- pypy/branch/2.5-features/pypy/module/_file/interp_file.py (original) +++ pypy/branch/2.5-features/pypy/module/_file/interp_file.py Fri Jul 4 20:37:22 2008 @@ -78,7 +78,7 @@ def direct___enter__(self): return self - def direct___exit__(self, excinfo): + def direct___exit__(self, __args__): self.direct_close() # can't return close() value return None @@ -271,10 +271,10 @@ _decl(locals(), "__init__", ['self', str, str, int], """Opens a file.""") - _decl(locals(), "__enter__", ['self'], """enter__() -> self.""") + _decl(locals(), "__enter__", ['self'], """__enter__() -> self.""") _decl(locals(), "__exit__", ['self', Arguments], - """exit__(*excinfo) -> None. Closes the file.""") + """__exit__(*excinfo) -> None. Closes the file.""") _decl(locals(), "close", ['self'], """close() -> None or (perhaps) an integer. Close the file. From bgola at codespeak.net Fri Jul 4 22:54:30 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Fri, 4 Jul 2008 22:54:30 +0200 (CEST) Subject: [pypy-svn] r56318 - pypy/branch/2.5-features/pypy/interpreter/pyparser/data Message-ID: <20080704205430.B666F2A8002@codespeak.net> Author: bgola Date: Fri Jul 4 22:54:30 2008 New Revision: 56318 Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 Log: (temp?) change to yield_stmt Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyparser/data/Grammar2.5 Fri Jul 4 22:54:30 2008 @@ -58,7 +58,7 @@ break_stmt: 'break' continue_stmt: 'continue' return_stmt: 'return' [testlist] -yield_stmt: yield_expr +yield_stmt: 'yield' [testlist] raise_stmt: 'raise' [test [',' test [',' test]]] import_stmt: import_name | import_from import_name: 'import' dotted_as_names From fijal at codespeak.net Sat Jul 5 12:44:59 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Sat, 5 Jul 2008 12:44:59 +0200 (CEST) Subject: [pypy-svn] r56324 - pypy/extradoc/talk/ep2008 Message-ID: <20080705104459.A1076168033@codespeak.net> Author: fijal Date: Sat Jul 5 12:44:56 2008 New Revision: 56324 Added: pypy/extradoc/talk/ep2008/nursery.fig pypy/extradoc/talk/ep2008/semispace.fig Modified: pypy/extradoc/talk/ep2008/status.txt Log: Check in two pictures and stuff I've been working on a plane before I go to bed Added: pypy/extradoc/talk/ep2008/nursery.fig ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/nursery.fig Sat Jul 5 12:44:56 2008 @@ -0,0 +1,40 @@ +#FIG 3.2 Produced by xfig version 3.2.5 +Landscape +Center +Metric +A4 +100.00 +Single +-2 +1200 2 +2 2 0 1 0 11 50 -1 -1 0.000 0 0 -1 0 0 5 + 3260 2720 7100 2720 7100 4370 3260 4370 3260 2720 +2 2 0 1 0 11 50 -1 -1 0.000 0 0 -1 0 0 5 + 3240 4590 7110 4590 7110 4860 3240 4860 3240 4590 +2 1 0 1 0 11 50 -1 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 4140 5490 4140 4860 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 3240 4590 3420 4590 3420 4860 3240 4860 3240 4590 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 3420 4590 3780 4590 3780 4860 3420 4860 3420 4590 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 3780 4590 3870 4590 3870 4860 3780 4860 3780 4590 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 3870 4590 4140 4590 4140 4860 3870 4860 3870 4590 +3 0 0 1 0 11 50 -1 -1 0.000 0 1 0 4 + 0 0 1.00 60.00 120.00 + 2340 5490 2700 5130 3420 5130 3690 4860 + 0.000 1.000 1.000 0.000 +3 0 1 1 0 11 50 -1 -1 4.000 0 1 0 3 + 0 0 1.00 60.00 120.00 + 4230 3780 3600 4140 3600 4590 + 0.000 1.000 0.000 +3 0 1 1 0 11 50 -1 -1 4.000 0 1 0 3 + 0 0 1.00 60.00 120.00 + 5310 3870 4320 4230 3960 4590 + 0.000 1.000 0.000 +4 0 0 50 -1 0 12 0.0000 4 195 1965 3995 3515 Some GC (semispace)\001 +4 0 0 50 -1 0 12 0.0000 4 195 720 4410 5130 Nursery\001 +4 0 0 50 -1 0 12 0.0000 4 195 2025 3690 5670 Current position pointer\001 +4 0 0 50 -1 0 12 0.0000 4 195 1515 1710 5670 Allocated objects\001 Added: pypy/extradoc/talk/ep2008/semispace.fig ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/semispace.fig Sat Jul 5 12:44:56 2008 @@ -0,0 +1,47 @@ +#FIG 3.2 Produced by xfig version 3.2.5 +Landscape +Center +Metric +A4 +100.00 +Single +-2 +1200 2 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 1350 1980 3150 1980 3150 5040 1350 5040 1350 1980 +2 2 0 1 0 7 50 -1 -1 0.000 0 0 -1 0 0 5 + 4050 1980 5850 1980 5850 5040 4050 5040 4050 1980 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 1350 1980 3150 1980 3150 2250 1350 2250 1350 1980 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 1350 2520 3150 2520 3150 2790 1350 2790 1350 2520 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 1350 3690 3150 3690 3150 3960 1350 3960 1350 3690 +2 2 0 1 0 11 50 -1 20 0.000 0 0 -1 0 0 5 + 1350 4500 3150 4500 3150 4770 1350 4770 1350 4500 +2 2 0 1 0 11 50 -1 32 0.000 0 0 -1 0 0 5 + 4050 1980 5850 1980 5850 2250 4050 2250 4050 1980 +2 2 0 1 0 11 50 -1 32 0.000 0 0 -1 0 0 5 + 4050 2250 5850 2250 5850 2520 4050 2520 4050 2250 +2 2 0 1 0 11 50 -1 32 0.000 0 0 -1 0 0 5 + 4050 2520 5850 2520 5850 2790 4050 2790 4050 2520 +2 2 0 1 0 11 50 -1 32 0.000 0 0 -1 0 0 5 + 4050 2790 5850 2790 5850 3060 4050 3060 4050 2790 +3 0 0 1 0 11 50 -1 -1 0.000 0 1 0 2 + 0 0 1.00 60.00 120.00 + 3152 2114 4040 2114 + 0.000 0.000 +3 0 0 1 0 11 50 -1 -1 0.000 0 1 0 4 + 0 0 1.00 60.00 120.00 + 3122 2654 3602 2654 3602 2420 4046 2420 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 11 50 -1 -1 0.000 0 1 0 4 + 0 0 1.00 60.00 120.00 + 3146 3866 3602 3434 3590 2948 4046 2660 + 0.000 1.000 1.000 0.000 +3 0 0 1 0 11 50 -1 -1 0.000 0 1 0 4 + 0 0 1.00 60.00 120.00 + 3146 4640 3560 4274 3776 3386 4046 2948 + 0.000 1.000 1.000 0.000 +4 0 0 50 -1 0 12 0.0000 4 195 1215 1710 1800 1st semispace\001 +4 0 0 50 -1 0 12 0.0000 4 195 1275 4230 1800 2nd semispace\001 Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Sat Jul 5 12:44:56 2008 @@ -1,6 +1,6 @@ -=========== -PyPy status -=========== +================ +PyPy status talk +================ :Author: Maciej Fijalkowski :Location: Europython 2008 @@ -20,87 +20,96 @@ Production ready ===================== -* We worked a lot on running +* we worked a lot on running existing applications on top of PyPy -* Sometimes requiring to change applications slightly +* sometimes requiring to change applications slightly + +* especially refcounting details tend to be a problem + +:: -* Especially refcounting details tend to be a problem open('xxx', 'w').write('stuff') CTypes ====== -* Official way to have bindings to +* official way to have bindings to external (C) libraries for PyPy -* Slow, but getting better +* slow, but getting better -* Sponsored by Google +* can handle i.e. pysqlite-ctypes, pyglet, pymunk or Sole Scion -* Can handle i.e. pysqlite-ctypes, pyglet, pymunk or Sole Scion +* ctypes is getting better as a side effect: -XXX demo of sole scion and pyglet + * errno handling -* ctypes is getting better as a side effect: + * bugfixes - * bugs found - * helper libraries - * possibility of porting to Jython/IronPython + * helper libraries + +* part of google sponsoring + +* demo CTypes configure ================ -* Our own small addition to general +* our own small addition to general CTypes usefulness -* Invokes C compiler for small details +* invokes C compiler for small details -* Can handle #defines, types, structure layout +* can handle #defines, types, structure layout etc. -XXX demo - Sqlite ====== -* Part of cpython stdlib since 2.5 +* part of cpython stdlib since 2.5 -* We use Gerhard Haering's CTypes version +* we use Gerhard Haering's CTypes version -* Works reasonably well after some fixes +* works reasonably well after some fixes Django ====== -* We run (almost) unmodified Django +* we run (almost) unmodified Django -* Only sqlite DB backend for now +* only sqlite DB backend for now -* Cooperation with Django people to make sure that Django 1.0 works with PyPy +* cooperation with Django people to make sure + that Django 1.0 works with PyPy -XXX demo +XXX demo, link Pylons ====== -* Worked almost out of the box once eggs - are working (1 day) +* worked almost out of the box once eggs + were working (1 day) -* No SQLAlchemy yet, obscure problems +* no SQLAlchemy yet, obscure problems ahead -XXX demo +* unmodified passes all tests + +XXX demo, link Twisted & Nevow =============== -* Twisted have some glitches +* twisted have some glitches, but mostly works + +* nevow works -* Nevow works +* we don't support PyCrypto nor PyOpenSSL and we + won't anytime soon (if nobody contributes CTypes or rpython + versions) -* We don't support PyCrypto nor PyOpenSSL and we - won't anytime soon (if nobody contributes CTypes versions) +XXX link Other software ============== @@ -117,7 +126,7 @@ * non-string keys in __dict__ of types -* list comprehension variable in __all__ +* exact naming of a list comprehension variable * relying on untested and undocumented private stuff (zipimport._zip_directory_cache) @@ -125,92 +134,108 @@ * exact message matching in exception catching code +* refcounting details + Conclusion on Compatibility ============================ * lessons learned: There is no feature obscure enough for people - not to rely on it. But PyPy can usually mimick CPython sufficiently. + not to rely on it. But PyPy can usually mimick CPython sufficiently. * pypy-c probably most compatible to CPython Interpreter * main blocker for running apps will be missing external modules - Speed - comparison with CPython =============================== -* We're something between 0.8-2x slower than +* we're something between 0.8-2x slower than CPython on various benchmarks. * gcbench - 0.8 (because of our faster GC) -* steady but slow progress +* steady but slow progress + +* we hope for our JIT to be a huge leap ahead Speed - decent GCs ================== -* Faster than refcounting +* faster than refcounting -* Better handling of obscure cases +* better handling of unusual patterns -* Troubles with for example communication with C +* troubles with for example communication with C + +* details on different talk Speed - JIT generator ===================== -* Not ready yet! +* not ready yet! + +* will be super fast -* Will be super fast +* prolog prototype -* Prolog prototype +* psyco is a nice proof that this approach + would work Other backends ============== * PyPy-jvm runs! -* More integration between pypy-cli and .NET +* more integration between pypy-cli and .NET -* General speed improvements +* general speed improvements -* Both backends are progressing - very slowly though +* both backends are progressing - very slowly though * contributors wanted! Sandboxing ========== -* Fully sandboxed python interpreter +* fully sandboxed python interpreter -* All external calls to C goes via another +* all external calls to C goes via another python process -* Special library for making custom +* special library for making custom policies -XXX image +.. image:: sandboxed.png + :scale: 30 + :align: center XXX demo -Cleanup sprint -============== +A lot of cleanups +================= -* Got rid of semi-cool semi-working +* got rid of semi-cool semi-working proofs of concept -* Reduced code size +* examples: CPython extension compiler, + rctypes -* Allowed us to progress forward into +* reduced code size + +* allowed us to progress forward into advanced features +* included sprint dedicated to cleanup + (on which noone was allowed to add features) + Plans ===== -* More JIT - faster Python +* more JIT - faster Python -* Support full CPython's stdlib +* support full CPython's stdlib -* Aim for more funding in pushing pypy +* aim for more funding in pushing pypy forward (more at the next talk) Links From hpk at codespeak.net Sat Jul 5 12:46:42 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sat, 5 Jul 2008 12:46:42 +0200 (CEST) Subject: [pypy-svn] r56325 - pypy/dist/pypy/rpython/memory/gc Message-ID: <20080705104642.518D12A805E@codespeak.net> Author: hpk Date: Sat Jul 5 12:46:41 2008 New Revision: 56325 Modified: pypy/dist/pypy/rpython/memory/gc/generation.py Log: fixes sandboxing with generational gc (hint from armin) Modified: pypy/dist/pypy/rpython/memory/gc/generation.py ============================================================================== --- pypy/dist/pypy/rpython/memory/gc/generation.py (original) +++ pypy/dist/pypy/rpython/memory/gc/generation.py Sat Jul 5 12:46:41 2008 @@ -614,7 +614,8 @@ sysctlbyname = rffi.llexternal('sysctlbyname', [rffi.CCHARP, rffi.VOIDP, rffi.SIZE_TP, rffi.VOIDP, rffi.SIZE_T], - rffi.INT) + rffi.INT, + sandboxsafe=True) def estimate_best_nursery_size(): """Try to estimate the best nursery size at run-time, depending From antocuni at codespeak.net Sat Jul 5 14:32:45 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Sat, 5 Jul 2008 14:32:45 +0200 (CEST) Subject: [pypy-svn] r56326 - pypy/dist/pypy/translator Message-ID: <20080705123245.A0EB416A0C5@codespeak.net> Author: antocuni Date: Sat Jul 5 14:32:44 2008 New Revision: 56326 Modified: pypy/dist/pypy/translator/driver.py Log: follow symlink when searching for *-cli-data and *-jvm.jar Modified: pypy/dist/pypy/translator/driver.py ============================================================================== --- pypy/dist/pypy/translator/driver.py (original) +++ pypy/dist/pypy/translator/driver.py Sat Jul 5 14:32:44 2008 @@ -679,8 +679,13 @@ f = file(newexename, 'w') f.write("""#!/bin/bash LEDIT=`type -p ledit` +EXE=`readlink $0` +if [ -z $EXE ] +then + EXE=$0 +fi if uname -s | grep -iq Cygwin ; then MONO=; else MONO=mono; fi -$LEDIT $MONO "$(dirname $0)/$(basename $0)-data/%s" "$@" # XXX doesn't work if it's placed in PATH +$LEDIT $MONO "$(dirname $EXE)/$(basename $EXE)-data/%s" "$@" # XXX doesn't work if it's placed in PATH """ % main_exe_name) f.close() os.chmod(newexename, 0755) @@ -751,7 +756,12 @@ f = file(newexename, 'w') f.write("""#!/bin/bash LEDIT=`type -p ledit` -$LEDIT java -Xmx256m -jar $0.jar "$@" +EXE=`readlink $0` +if [ -z $EXE ] +then + EXE=$0 +fi +$LEDIT java -Xmx256m -jar $EXE.jar "$@" """) f.close() os.chmod(newexename, 0755) From pypy-svn at codespeak.net Sun Jul 6 01:54:44 2008 From: pypy-svn at codespeak.net (VIAGRA ® Official Site) Date: Sun, 6 Jul 2008 01:54:44 +0200 (CEST) Subject: [pypy-svn] SPECIAL OFFERS-Pharmacy Online July Sale 70% Message-ID: <20080706025237.19698.qmail@dtmd-4db210d2.pool.einsundeins.de> An HTML attachment was scrubbed... URL: From fijal at codespeak.net Sun Jul 6 05:05:52 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Sun, 6 Jul 2008 05:05:52 +0200 (CEST) Subject: [pypy-svn] r56331 - pypy/dist/pypy/translator/js Message-ID: <20080706030552.7956D2A8097@codespeak.net> Author: fijal Date: Sun Jul 6 05:05:50 2008 New Revision: 56331 Modified: pypy/dist/pypy/translator/js/function.py Log: Adapt a bit to new interface. Not sure how to implement classof opcode in js Modified: pypy/dist/pypy/translator/js/function.py ============================================================================== --- pypy/dist/pypy/translator/js/function.py (original) +++ pypy/dist/pypy/translator/js/function.py Sun Jul 6 05:05:50 2008 @@ -1,7 +1,3 @@ -try: - set -except NameError: - from sets import Set as set from pypy.objspace.flow import model as flowmodel from pypy.rpython.lltypesystem.lltype import Signed, Unsigned, Void, Bool, Float @@ -221,7 +217,7 @@ pass def begin_catch(self, llexitcase): - real_name = self.cts.lltype_to_cts(llexitcase._inst.class_._INSTANCE) + real_name = self.cts.lltype_to_cts(llexitcase._INSTANCE) s = "isinstanceof(exc, %s)"%real_name self.ilasm.branch_if_string(s) From pedronis at codespeak.net Sun Jul 6 05:10:03 2008 From: pedronis at codespeak.net (pedronis at codespeak.net) Date: Sun, 6 Jul 2008 05:10:03 +0200 (CEST) Subject: [pypy-svn] r56332 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080706031003.5946A16853F@codespeak.net> Author: pedronis Date: Sun Jul 6 05:10:01 2008 New Revision: 56332 Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt Log: my dates Modified: pypy/extradoc/sprintinfo/post-ep2008/people.txt ============================================================================== --- pypy/extradoc/sprintinfo/post-ep2008/people.txt (original) +++ pypy/extradoc/sprintinfo/post-ep2008/people.txt Sun Jul 6 05:10:01 2008 @@ -16,6 +16,7 @@ Jakub Gustak 6-13 http://hospitalityclub.org/ Stephan Diehl 6-13 http://www.litinterp.lt/ Armin Rigo 6-13 Ambassador Hotel +Samuele Pedroni 6-13 conf hotel ==================== ============== ============================ People on the following list were present at previous sprints: @@ -23,7 +24,6 @@ ==================== ============== ===================== Name Arrive/Depart Accomodation ==================== ============== ===================== -Samuele Pedroni ? ? Carl Friedrich Bolz ? ? Alexander Schremmer ? ? Holger Krekel ? ? From fijal at codespeak.net Sun Jul 6 05:22:00 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Sun, 6 Jul 2008 05:22:00 +0200 (CEST) Subject: [pypy-svn] r56333 - pypy/dist/pypy/translator/js Message-ID: <20080706032200.7DF53168540@codespeak.net> Author: fijal Date: Sun Jul 6 05:21:59 2008 New Revision: 56333 Modified: pypy/dist/pypy/translator/js/metavm.py pypy/dist/pypy/translator/js/opcodes.py Log: Implement classof Modified: pypy/dist/pypy/translator/js/metavm.py ============================================================================== --- pypy/dist/pypy/translator/js/metavm.py (original) +++ pypy/dist/pypy/translator/js/metavm.py Sun Jul 6 05:21:59 2008 @@ -146,11 +146,14 @@ generator.get_field(None, field) class _GetPredefinedField(MicroInstruction): - def __init__(self, field): + def __init__(self, field, num=1): self.field = field + self.num = num def render(self, generator, op): - this = op.args[1] + if op.result.concretetype is ootype.Void: + return + this = op.args[self.num] generator.load(this) generator.get_field(None, self.field) Modified: pypy/dist/pypy/translator/js/opcodes.py ============================================================================== --- pypy/dist/pypy/translator/js/opcodes.py (original) +++ pypy/dist/pypy/translator/js/opcodes.py Sun Jul 6 05:21:59 2008 @@ -12,7 +12,7 @@ CopyName, CastString, _Prefix, _CastFun, _NotImplemented, CallBuiltin,\ CallBuiltinObject, GetBuiltinField, SetBuiltinField, IndirectCall,\ CallExternalObject, SetExternalField, _CastMethod, _LoadConst,\ - DiscardStack, CheckLength, fix_opcodes + DiscardStack, CheckLength, fix_opcodes, _GetPredefinedField from pypy.translator.js.jsbuiltin import Builtins from pypy.rpython.ootypesystem import ootype @@ -153,6 +153,8 @@ 'cast_float_to_uint': [PushAllArgs,_CastFun("Math.floor",1)], 'cast_float_to_longlong': [PushAllArgs,_CastFun("Math.floor",1)], 'truncate_longlong_to_int': CopyName, + + 'classof' : [_GetPredefinedField('_class', 0)], 'debug_assert' : DoNothing, 'resume_point' : DoNothing, From fijal at codespeak.net Sun Jul 6 18:00:12 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Sun, 6 Jul 2008 18:00:12 +0200 (CEST) Subject: [pypy-svn] r56348 - pypy/extradoc/talk/ep2008 Message-ID: <20080706160012.40E7B39B599@codespeak.net> Author: fijal Date: Sun Jul 6 18:00:09 2008 New Revision: 56348 Modified: pypy/extradoc/talk/ep2008/status.txt Log: Update the talk a bit Modified: pypy/extradoc/talk/ep2008/status.txt ============================================================================== --- pypy/extradoc/talk/ep2008/status.txt (original) +++ pypy/extradoc/talk/ep2008/status.txt Sun Jul 6 18:00:09 2008 @@ -1,10 +1,6 @@ -================ +====================== PyPy status talk -================ - -:Author: Maciej Fijalkowski -:Location: Europython 2008 -:Date: 7 July 2008 +====================== What this talk is about ======================= @@ -83,7 +79,7 @@ * cooperation with Django people to make sure that Django 1.0 works with PyPy -XXX demo, link +* http://www.djangoproject.com/ Pylons ====== @@ -96,7 +92,7 @@ * unmodified passes all tests -XXX demo, link +* http://pylonshq.com/ Twisted & Nevow =============== @@ -109,7 +105,7 @@ won't anytime soon (if nobody contributes CTypes or rpython versions) -XXX link +* http://twistedmatrix.com/ Other software ============== @@ -121,8 +117,8 @@ * various smaller things, templating engines, most pure-python software -Obscure differences to CPython -============================== +Obscure details that people rely on +=================================== * non-string keys in __dict__ of types @@ -209,8 +205,6 @@ :scale: 30 :align: center -XXX demo - A lot of cleanups ================= From bgola at codespeak.net Mon Jul 7 15:30:52 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Mon, 7 Jul 2008 15:30:52 +0200 (CEST) Subject: [pypy-svn] r56350 - pypy/branch/2.5-features/pypy/module/symbol Message-ID: <20080707133052.8B03B2D8008@codespeak.net> Author: bgola Date: Mon Jul 7 15:30:50 2008 New Revision: 56350 Modified: pypy/branch/2.5-features/pypy/module/symbol/__init__.py Log: removing unuseful(?) line (hardcoded reference to 2.4) Modified: pypy/branch/2.5-features/pypy/module/symbol/__init__.py ============================================================================== --- pypy/branch/2.5-features/pypy/module/symbol/__init__.py (original) +++ pypy/branch/2.5-features/pypy/module/symbol/__init__.py Mon Jul 7 15:30:50 2008 @@ -38,6 +38,3 @@ Module.interpleveldefs[name] = 'space.wrap(%d)' % val sym_name[val] = name Module.interpleveldefs['sym_name'] = 'space.wrap(%r)' % (sym_name,) - -# This is very evil -_init_symbols('2.4') From cfbolz at codespeak.net Mon Jul 7 15:53:15 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 15:53:15 +0200 (CEST) Subject: [pypy-svn] r56351 - in pypy/dist/pypy/module/posix: . test Message-ID: <20080707135315.88BEF2A00DF@codespeak.net> Author: cfbolz Date: Mon Jul 7 15:53:15 2008 New Revision: 56351 Modified: pypy/dist/pypy/module/posix/interp_posix.py pypy/dist/pypy/module/posix/test/test_posix2.py Log: rewrap OSError in setuid Modified: pypy/dist/pypy/module/posix/interp_posix.py ============================================================================== --- pypy/dist/pypy/module/posix/interp_posix.py (original) +++ pypy/dist/pypy/module/posix/interp_posix.py Mon Jul 7 15:53:15 2008 @@ -572,7 +572,10 @@ Set the current process's user id. """ - os.setuid(arg) + try: + os.setuid(arg) + except OSError, e: + raise wrap_oserror(space, e) return space.w_None setuid.unwrap_spec = [ObjSpace, int] Modified: pypy/dist/pypy/module/posix/test/test_posix2.py ============================================================================== --- pypy/dist/pypy/module/posix/test/test_posix2.py (original) +++ pypy/dist/pypy/module/posix/test/test_posix2.py Mon Jul 7 15:53:15 2008 @@ -306,6 +306,11 @@ assert os.getuid() == self.getuid assert os.geteuid() == self.geteuid + if hasattr(os, 'setuid'): + def test_os_setuid_error(self): + os = self.posix + raises(OSError, os.setuid, -100000) + if hasattr(os, 'getgid'): def test_os_getgid(self): os = self.posix From cfbolz at codespeak.net Mon Jul 7 16:03:50 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 16:03:50 +0200 (CEST) Subject: [pypy-svn] r56352 - pypy/dist/pypy/module/_pickle_support Message-ID: <20080707140350.07D50698038@codespeak.net> Author: cfbolz Date: Mon Jul 7 16:03:50 2008 New Revision: 56352 Modified: pypy/dist/pypy/module/_pickle_support/maker.py Log: Strange, strange code in _pickle_support. It looks wrong somehow. Fix at least generator_new, which used to be crashable. frame_new is very strange still. Modified: pypy/dist/pypy/module/_pickle_support/maker.py ============================================================================== --- pypy/dist/pypy/module/_pickle_support/maker.py (original) +++ pypy/dist/pypy/module/_pickle_support/maker.py Mon Jul 7 16:03:50 2008 @@ -55,30 +55,26 @@ w_len = space.len(w_seq) index = space.int_w(w_index) - space.int_w(w_len) return W_ReverseSeqIterObject(space, w_seq, index) - def frame_new(space, __args__): args_w, kwds_w = __args__.unpack() w_pycode, = args_w pycode = space.interp_w(PyCode, w_pycode) w = space.wrap + # XXX I don't get this? why isn't pycode actually used for anything? new_frame = instantiate(space.FrameClass) # XXX fish return space.wrap(new_frame) -frame_new.unwrap_spec = [ObjSpace, Arguments] +frame_new.unwrap_spec = [ObjSpace, PyCode] def traceback_new(space): tb = instantiate(PyTraceback) return space.wrap(tb) traceback_new.unwrap_spec = [ObjSpace] -def generator_new(space, __args__): - args_w, kwds_w = __args__.unpack() #stolen from std/fake.py - w_frame, w_running = args_w - frame = space.interp_w(PyFrame, w_frame) - running = space.int_w(w_running) +def generator_new(space, frame, running): new_generator = GeneratorIterator(frame) new_generator.running = running return space.wrap(new_generator) -generator_new.unwrap_spec = [ObjSpace, Arguments] +generator_new.unwrap_spec = [ObjSpace, PyFrame, int] def xrangeiter_new(space, current, remaining, step): from pypy.module.__builtin__.functional import W_XRangeIterator From cfbolz at codespeak.net Mon Jul 7 16:11:56 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 16:11:56 +0200 (CEST) Subject: [pypy-svn] r56353 - in pypy/dist/pypy/module/posix: . test Message-ID: <20080707141156.03DD3698048@codespeak.net> Author: cfbolz Date: Mon Jul 7 16:11:55 2008 New Revision: 56353 Modified: pypy/dist/pypy/module/posix/interp_posix.py pypy/dist/pypy/module/posix/test/test_posix2.py Log: setgid has the same problem Modified: pypy/dist/pypy/module/posix/interp_posix.py ============================================================================== --- pypy/dist/pypy/module/posix/interp_posix.py (original) +++ pypy/dist/pypy/module/posix/interp_posix.py Mon Jul 7 16:11:55 2008 @@ -584,7 +584,10 @@ Set the current process's group id. """ - os.setgid(arg) + try: + os.setgid(arg) + except OSError, e: + raise wrap_oserror(space, e) return space.w_None setgid.unwrap_spec = [ObjSpace, int] Modified: pypy/dist/pypy/module/posix/test/test_posix2.py ============================================================================== --- pypy/dist/pypy/module/posix/test/test_posix2.py (original) +++ pypy/dist/pypy/module/posix/test/test_posix2.py Mon Jul 7 16:11:55 2008 @@ -316,6 +316,11 @@ os = self.posix assert os.getgid() == self.getgid + if hasattr(os, 'setgid'): + def test_os_setgid_error(self): + os = self.posix + raises(OSError, os.setgid, -100000) + if hasattr(os, 'sysconf'): def test_os_sysconf(self): os = self.posix From cfbolz at codespeak.net Mon Jul 7 16:31:04 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 16:31:04 +0200 (CEST) Subject: [pypy-svn] r56354 - in pypy/dist/pypy/module/posix: . test Message-ID: <20080707143104.CB73616855D@codespeak.net> Author: cfbolz Date: Mon Jul 7 16:31:02 2008 New Revision: 56354 Modified: pypy/dist/pypy/module/posix/interp_posix.py pypy/dist/pypy/module/posix/test/test_posix2.py Log: A forgotten error-check in os.sysconf. Modified: pypy/dist/pypy/module/posix/interp_posix.py ============================================================================== --- pypy/dist/pypy/module/posix/interp_posix.py (original) +++ pypy/dist/pypy/module/posix/interp_posix.py Mon Jul 7 16:31:02 2008 @@ -630,8 +630,13 @@ ttyname.unwrap_spec = [ObjSpace, int] def sysconf(space, w_num_or_name): + # XXX slightly non-nice, reuses the sysconf of the underlying os module if space.is_true(space.isinstance(w_num_or_name, space.w_basestring)): - num = os.sysconf_names[space.str_w(w_num_or_name)] + try: + num = os.sysconf_names[space.str_w(w_num_or_name)] + except KeyError: + raise OperationError(space.w_ValueError, + space.wrap("unrecognized configuration name")) else: num = space.int_w(w_num_or_name) return space.wrap(os.sysconf(num)) Modified: pypy/dist/pypy/module/posix/test/test_posix2.py ============================================================================== --- pypy/dist/pypy/module/posix/test/test_posix2.py (original) +++ pypy/dist/pypy/module/posix/test/test_posix2.py Mon Jul 7 16:31:02 2008 @@ -328,6 +328,10 @@ assert os.sysconf(self.sysconf_name) == self.sysconf_result assert os.sysconf_names[self.sysconf_name] == self.sysconf_value + def test_os_sysconf_error(self): + os = self.posix + raises(ValueError, os.sysconf, "!@#$%!#$!@#") + def test_largefile(self): os = self.posix fd = os.open(self.path2 + 'test_largefile', os.O_RDWR | os.O_CREAT, 0666) From bgola at codespeak.net Mon Jul 7 16:56:46 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Mon, 7 Jul 2008 16:56:46 +0200 (CEST) Subject: [pypy-svn] r56355 - in pypy/branch/2.5-features/pypy/interpreter: . astcompiler pyparser test Message-ID: <20080707145646.E34B7169E37@codespeak.net> Author: bgola Date: Mon Jul 7 16:56:45 2008 New Revision: 56355 Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py pypy/branch/2.5-features/pypy/interpreter/generator.py pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py pypy/branch/2.5-features/pypy/interpreter/typedef.py Log: Supporting part of pep-342, generator stuff (.send()). Things may be broken now because of the current default magic number (see execute_generator_frame() in pyframe.py). Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py Mon Jul 7 16:56:45 2008 @@ -519,7 +519,7 @@ 'PRINT_ITEM_TO': -2, 'PRINT_NEWLINE': 0, 'PRINT_NEWLINE_TO': -1, - 'YIELD_VALUE': -1, + 'YIELD_VALUE': 0, 'EXEC_STMT': -3, 'BUILD_CLASS': -2, 'STORE_NAME': -1, Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py Mon Jul 7 16:56:45 2008 @@ -695,6 +695,7 @@ node.expr.accept( self ) self.emit('YIELD_VALUE') + self.emit('POP_TOP') for start, cont, anchor in stack: if cont: Modified: pypy/branch/2.5-features/pypy/interpreter/generator.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/generator.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/generator.py Mon Jul 7 16:56:45 2008 @@ -28,18 +28,26 @@ """x.__iter__() <==> iter(x)""" return self.space.wrap(self) - def descr_next(self): - """x.next() -> the next value, or raise StopIteration""" + def descr_send(self, w_arg=None): + """send(arg) -> send 'arg' into generator, +return next yielded value or raise StopIteration.""" space = self.space if self.running: raise OperationError(space.w_ValueError, space.wrap('generator already executing')) if self.frame.frame_finished_execution: - raise OperationError(space.w_StopIteration, space.w_None) + raise OperationError(space.w_StopIteration, space.w_None) + if self.frame.last_instr == -1: + if w_arg and not space.is_w(w_arg, space.w_None): + msg = "can't send non-None value to a just-started generator" + raise OperationError(space.w_TypeError, space.wrap(msg)) + else: + if not w_arg: + w_arg = space.w_None self.running = True try: try: - w_result = self.frame.execute_generator_frame(space.w_None) + w_result = self.frame.execute_generator_frame(w_arg) except OperationError: # errors finish a frame self.frame.frame_finished_execution = True @@ -52,3 +60,9 @@ finally: self.frame.f_back = None self.running = False + + def descr_next(self): + """x.next() -> the next value, or raise StopIteration""" + return self.descr_send() + + Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py Mon Jul 7 16:56:45 2008 @@ -880,6 +880,11 @@ def build_yield_stmt(builder, nb): atoms = get_atoms(builder, nb) + lineno = atoms[0].lineno + builder.push(ast.Discard(ast.Yield(atoms[1], lineno), lineno)) + +def build_yield_expr(builder, nb): + atoms = get_atoms(builder, nb) builder.push(ast.Yield(atoms[1], atoms[0].lineno)) def build_continue_stmt(builder, nb): @@ -1038,6 +1043,7 @@ 'import_name' : build_import_name, 'import_from' : build_import_from, 'yield_stmt' : build_yield_stmt, + 'yield_expr' : build_yield_expr, 'continue_stmt' : build_continue_stmt, 'del_stmt' : build_del_stmt, 'assert_stmt' : build_assert_stmt, Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py Mon Jul 7 16:56:45 2008 @@ -26,6 +26,14 @@ g = f() assert [x for x in g] == [1] + def test_generator5(self): + def f(): + v = (yield ) + yield v + g = f() + g.next() + assert g.send(42) == 42 + def test_generator_explicit_stopiteration(self): def f(): yield 1 Modified: pypy/branch/2.5-features/pypy/interpreter/typedef.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/typedef.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/typedef.py Mon Jul 7 16:56:45 2008 @@ -817,6 +817,8 @@ unwrap_spec=['self', ObjSpace]), next = interp2app(GeneratorIterator.descr_next, descrmismatch='next'), + send = interp2app(GeneratorIterator.descr_send, + descrmismatch='send'), __iter__ = interp2app(GeneratorIterator.descr__iter__, descrmismatch='__iter__'), gi_running = interp_attrproperty('running', cls=GeneratorIterator), From cfbolz at codespeak.net Mon Jul 7 17:30:49 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 17:30:49 +0200 (CEST) Subject: [pypy-svn] r56356 - in pypy/dist/pypy/module/recparser: . test Message-ID: <20080707153049.CF702169F21@codespeak.net> Author: cfbolz Date: Mon Jul 7 17:30:47 2008 New Revision: 56356 Modified: pypy/dist/pypy/module/recparser/pyparser.py pypy/dist/pypy/module/recparser/test/test_parser.py Log: Fix a segfault (IndexError) in the recparser. Modified: pypy/dist/pypy/module/recparser/pyparser.py ============================================================================== --- pypy/dist/pypy/module/recparser/pyparser.py (original) +++ pypy/dist/pypy/module/recparser/pyparser.py Mon Jul 7 17:30:47 2008 @@ -149,6 +149,10 @@ totuple = interp2app(STType.descr_totuple), ) +def get(space, name): + w_module = space.getbuiltinmodule('parser') + return space.getattr(w_module, space.wrap(name)) + def get_ast_compiler(space): from pypy.interpreter.pycompiler import PythonAstCompiler compiler = space.createcompiler() @@ -189,10 +193,15 @@ ast2tuple.unwrap_spec = [ObjSpace, STType, int] +def check_length(space, items, length): + if len(items) < length: + raise OperationError(get(space, "ParserError"), + space.wrap("argument too small")) def unwrap_syntax_tree( space, w_sequence ): items = space.unpackiterable( w_sequence ) parser = space.default_compiler.parser + check_length(space, items, 1) nodetype = space.int_w( items[0] ) if parser.is_base_token(nodetype): nodes = [] Modified: pypy/dist/pypy/module/recparser/test/test_parser.py ============================================================================== --- pypy/dist/pypy/module/recparser/test/test_parser.py (original) +++ pypy/dist/pypy/module/recparser/test/test_parser.py Mon Jul 7 17:30:47 2008 @@ -20,3 +20,11 @@ import parser parser.suite("a = 3").totuple() +class AppTestRecparserErrors: + def setup_class(cls): + cls.space = space + + def test_sequence2st_bug1(self): + import parser + raises(parser.ParserError, parser.sequence2st, ()) + From cfbolz at codespeak.net Mon Jul 7 18:44:59 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 18:44:59 +0200 (CEST) Subject: [pypy-svn] r56357 - in pypy/dist/pypy/module/termios: . test Message-ID: <20080707164459.D65A1169EA4@codespeak.net> Author: cfbolz Date: Mon Jul 7 18:44:57 2008 New Revision: 56357 Modified: pypy/dist/pypy/module/termios/interp_termios.py pypy/dist/pypy/module/termios/test/test_termios.py Log: Fix an RPython-ValueError in tcsetattr. Modified: pypy/dist/pypy/module/termios/interp_termios.py ============================================================================== --- pypy/dist/pypy/module/termios/interp_termios.py (original) +++ pypy/dist/pypy/module/termios/interp_termios.py Mon Jul 7 18:44:57 2008 @@ -28,34 +28,31 @@ return OperationError(w_exception_class, w_exception) def tcsetattr(space, fd, when, w_attributes): + from pypy.interpreter.baseobjspace import UnpackValueError try: - w_cc = space.getitem(w_attributes, space.wrap(-1)) - tup_w = space.getitem(w_attributes, space.newslice( - space.wrap(0), space.wrap(-1), space.wrap(1))) - w_iflag, w_oflag, w_cflag, w_lflag, w_ispeed, w_ospeed = \ - space.unpackiterable(tup_w) - w_builtin = space.getbuiltinmodule('__builtin__') - cc = [] - for w_c in space.unpackiterable(w_cc): - if space.is_true(space.isinstance(w_c, space.w_int)): - ch = space.call_function(space.getattr(w_builtin, - space.wrap('chr')), w_c) - cc.append(space.str_w(ch)) - else: - cc.append(space.str_w(w_c)) - tup = (space.int_w(w_iflag), space.int_w(w_oflag), - space.int_w(w_cflag), space.int_w(w_lflag), - space.int_w(w_ispeed), space.int_w(w_ospeed), cc) - try: - rtermios.tcsetattr(fd, when, tup) - except termios.error, e: - e.errno = e.args[0] - raise convert_error(space, e) - except OperationError, e: - if not e.match(space, space.w_TypeError): - raise - msg = "tcsetattr must be called with int, int and 7-arg tuple" - raise OperationError(space.w_TypeError, space.wrap(str(msg))) + w_iflag, w_oflag, w_cflag, w_lflag, w_ispeed, w_ospeed, w_cc = \ + space.unpackiterable(w_attributes, expected_length=7) + except UnpackValueError, e: + raise OperationError( + space.w_TypeError, + space.wrap("tcsetattr, arg 3: must be 7 element list")) + w_builtin = space.getbuiltinmodule('__builtin__') + cc = [] + for w_c in space.unpackiterable(w_cc): + if space.is_true(space.isinstance(w_c, space.w_int)): + ch = space.call_function(space.getattr(w_builtin, + space.wrap('chr')), w_c) + cc.append(space.str_w(ch)) + else: + cc.append(space.str_w(w_c)) + tup = (space.int_w(w_iflag), space.int_w(w_oflag), + space.int_w(w_cflag), space.int_w(w_lflag), + space.int_w(w_ispeed), space.int_w(w_ospeed), cc) + try: + rtermios.tcsetattr(fd, when, tup) + except termios.error, e: + e.errno = e.args[0] + raise convert_error(space, e) tcsetattr.unwrap_spec = [ObjSpace, int, int, W_Root] def tcgetattr(space, fd): Modified: pypy/dist/pypy/module/termios/test/test_termios.py ============================================================================== --- pypy/dist/pypy/module/termios/test/test_termios.py (original) +++ pypy/dist/pypy/module/termios/test/test_termios.py Mon Jul 7 18:44:57 2008 @@ -128,3 +128,6 @@ import termios raises(termios.error, "termios.tcgetattr(334)") + def test_error_tcsetattr(self): + import termios + raises(TypeError, termios.tcsetattr, 0, 1, (1, 2)) From cfbolz at codespeak.net Mon Jul 7 19:32:48 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 19:32:48 +0200 (CEST) Subject: [pypy-svn] r56358 - in pypy/dist/pypy/module/recparser: . test Message-ID: <20080707173248.B1CD0169F07@codespeak.net> Author: cfbolz Date: Mon Jul 7 19:32:45 2008 New Revision: 56358 Modified: pypy/dist/pypy/module/recparser/pyparser.py pypy/dist/pypy/module/recparser/test/test_parser.py Log: Wrap a parser error properly. Modified: pypy/dist/pypy/module/recparser/pyparser.py ============================================================================== --- pypy/dist/pypy/module/recparser/pyparser.py (original) +++ pypy/dist/pypy/module/recparser/pyparser.py Mon Jul 7 19:32:45 2008 @@ -222,8 +222,12 @@ def source2ast(space, source): + from pypy.interpreter.pyparser.error import SyntaxError compiler = get_ast_compiler(space) - return space.wrap(compiler.source2ast(source, 'exec')) + try: + return space.wrap(compiler.source2ast(source, 'exec')) + except SyntaxError, e: + raise OperationError(space.w_SyntaxError, e.wrap_info(space, "")) source2ast.unwrap_spec = [ObjSpace, str] Modified: pypy/dist/pypy/module/recparser/test/test_parser.py ============================================================================== --- pypy/dist/pypy/module/recparser/test/test_parser.py (original) +++ pypy/dist/pypy/module/recparser/test/test_parser.py Mon Jul 7 19:32:45 2008 @@ -28,3 +28,7 @@ import parser raises(parser.ParserError, parser.sequence2st, ()) + def test_source2ast_bug1(self): + import parser + raises(SyntaxError, parser.source2ast, "\xDE\xDA") + From cfbolz at codespeak.net Mon Jul 7 20:10:57 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 20:10:57 +0200 (CEST) Subject: [pypy-svn] r56359 - pypy/dist/pypy/module/_pickle_support Message-ID: <20080707181057.010AB169F09@codespeak.net> Author: cfbolz Date: Mon Jul 7 20:10:56 2008 New Revision: 56359 Modified: pypy/dist/pypy/module/_pickle_support/maker.py Log: Grr, nonsense. This stuff doesn't seem to be properly tested either. Modified: pypy/dist/pypy/module/_pickle_support/maker.py ============================================================================== --- pypy/dist/pypy/module/_pickle_support/maker.py (original) +++ pypy/dist/pypy/module/_pickle_support/maker.py Mon Jul 7 20:10:56 2008 @@ -63,7 +63,7 @@ # XXX I don't get this? why isn't pycode actually used for anything? new_frame = instantiate(space.FrameClass) # XXX fish return space.wrap(new_frame) -frame_new.unwrap_spec = [ObjSpace, PyCode] +frame_new.unwrap_spec = [ObjSpace, Arguments] def traceback_new(space): tb = instantiate(PyTraceback) From cfbolz at codespeak.net Mon Jul 7 20:54:46 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 20:54:46 +0200 (CEST) Subject: [pypy-svn] r56360 - in pypy/dist/pypy: interpreter module/_pickle_support Message-ID: <20080707185446.E8F3D169F00@codespeak.net> Author: cfbolz Date: Mon Jul 7 20:54:45 2008 New Revision: 56360 Modified: pypy/dist/pypy/interpreter/pycompiler.py pypy/dist/pypy/interpreter/pyframe.py pypy/dist/pypy/module/_pickle_support/maker.py Log: OK, things cleared up: At one point there were different frame classes and it was necessary to instantiate the right one, based one the code object. Since there is only one frame class at the moment, this argument can actually go away. Modified: pypy/dist/pypy/interpreter/pycompiler.py ============================================================================== --- pypy/dist/pypy/interpreter/pycompiler.py (original) +++ pypy/dist/pypy/interpreter/pycompiler.py Mon Jul 7 20:54:45 2008 @@ -227,7 +227,7 @@ self.compiler_flags = self.futureFlags.allowed_flags def compile(self, source, filename, mode, flags): - from pyparser.error import SyntaxError + from pypy.interpreter.pyparser.error import SyntaxError from pypy.interpreter import astcompiler from pypy.interpreter.astcompiler.pycodegen import ModuleCodeGenerator from pypy.interpreter.astcompiler.pycodegen import InteractiveCodeGenerator Modified: pypy/dist/pypy/interpreter/pyframe.py ============================================================================== --- pypy/dist/pypy/interpreter/pyframe.py (original) +++ pypy/dist/pypy/interpreter/pyframe.py Mon Jul 7 20:54:45 2008 @@ -255,10 +255,6 @@ w_blockstack = nt([block._get_state_(space) for block in self.blockstack]) w_fastlocals = maker.slp_into_tuple_with_nulls(space, self.fastlocals_w) - tup_base = [ - w(self.pycode), - ] - if self.last_exception is None: w_exc_value = space.w_None w_tb = space.w_None @@ -290,7 +286,7 @@ w_cells, ] - return nt([new_inst, nt(tup_base), nt(tup_state)]) + return nt([new_inst, nt([]), nt(tup_state)]) def descr__setstate__(self, space, w_args): from pypy.module._pickle_support import maker # helper fns Modified: pypy/dist/pypy/module/_pickle_support/maker.py ============================================================================== --- pypy/dist/pypy/module/_pickle_support/maker.py (original) +++ pypy/dist/pypy/module/_pickle_support/maker.py Mon Jul 7 20:54:45 2008 @@ -55,15 +55,11 @@ w_len = space.len(w_seq) index = space.int_w(w_index) - space.int_w(w_len) return W_ReverseSeqIterObject(space, w_seq, index) -def frame_new(space, __args__): - args_w, kwds_w = __args__.unpack() - w_pycode, = args_w - pycode = space.interp_w(PyCode, w_pycode) - w = space.wrap - # XXX I don't get this? why isn't pycode actually used for anything? + +def frame_new(space): new_frame = instantiate(space.FrameClass) # XXX fish return space.wrap(new_frame) -frame_new.unwrap_spec = [ObjSpace, Arguments] +frame_new.unwrap_spec = [ObjSpace] def traceback_new(space): tb = instantiate(PyTraceback) From cfbolz at codespeak.net Mon Jul 7 23:16:44 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 23:16:44 +0200 (CEST) Subject: [pypy-svn] r56361 - in pypy/dist/pypy/lib: . app_test Message-ID: <20080707211644.7581A2A00DE@codespeak.net> Author: cfbolz Date: Mon Jul 7 23:16:43 2008 New Revision: 56361 Modified: pypy/dist/pypy/lib/app_test/test_pyexpat.py pypy/dist/pypy/lib/pyexpat.py Log: Better error-checking for ParserCreate (before you could segfault pypy). Modified: pypy/dist/pypy/lib/app_test/test_pyexpat.py ============================================================================== --- pypy/dist/pypy/lib/app_test/test_pyexpat.py (original) +++ pypy/dist/pypy/lib/app_test/test_pyexpat.py Mon Jul 7 23:16:43 2008 @@ -643,3 +643,6 @@ assert parser.buffer_size == 1024 parser.Parse(xml2, 1) assert self.n == 4 + + def test_segfault(self): + py.test.raises(TypeError, expat.ParserCreate, 1234123123) Modified: pypy/dist/pypy/lib/pyexpat.py ============================================================================== --- pypy/dist/pypy/lib/pyexpat.py (original) +++ pypy/dist/pypy/lib/pyexpat.py Mon Jul 7 23:16:43 2008 @@ -436,6 +436,9 @@ xxx def ParserCreate(encoding=None, namespace_separator=None, intern=None): + if (not isinstance(encoding, str) and + not encoding is None): + raise TypeError("ParserCreate() argument 1 must be string or None, not %s" % encoding.__class__.__name__) if (not isinstance(namespace_separator, str) and not namespace_separator is None): raise TypeError("ParserCreate() argument 2 must be string or None, not %s" % namespace_separator.__class__.__name__) From cfbolz at codespeak.net Mon Jul 7 23:31:28 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Mon, 7 Jul 2008 23:31:28 +0200 (CEST) Subject: [pypy-svn] r56362 - in pypy/dist/pypy/module/recparser: . test Message-ID: <20080707213128.778E3698054@codespeak.net> Author: cfbolz Date: Mon Jul 7 23:31:27 2008 New Revision: 56362 Modified: pypy/dist/pypy/module/recparser/pyparser.py pypy/dist/pypy/module/recparser/test/test_parser.py Log: Another parser segfault. I knew why I gave the test method a number :-). Modified: pypy/dist/pypy/module/recparser/pyparser.py ============================================================================== --- pypy/dist/pypy/module/recparser/pyparser.py (original) +++ pypy/dist/pypy/module/recparser/pyparser.py Mon Jul 7 23:31:27 2008 @@ -210,6 +210,7 @@ nodes.append( node ) return SyntaxNode( nodetype, nodes ) else: + check_length(space, items, 2) value = space.str_w( items[1] ) lineno = -1 if len(items)>2: Modified: pypy/dist/pypy/module/recparser/test/test_parser.py ============================================================================== --- pypy/dist/pypy/module/recparser/test/test_parser.py (original) +++ pypy/dist/pypy/module/recparser/test/test_parser.py Mon Jul 7 23:31:27 2008 @@ -28,6 +28,10 @@ import parser raises(parser.ParserError, parser.sequence2st, ()) + def test_sequence2st_bug1(self): + import parser + raises(parser.ParserError, parser.sequence2st, (True,)) + def test_source2ast_bug1(self): import parser raises(SyntaxError, parser.source2ast, "\xDE\xDA") From haypo at codespeak.net Tue Jul 8 01:21:11 2008 From: haypo at codespeak.net (haypo at codespeak.net) Date: Tue, 8 Jul 2008 01:21:11 +0200 (CEST) Subject: [pypy-svn] r56364 - pypy/dist/pypy/lib Message-ID: <20080707232111.B9BCE16A11C@codespeak.net> Author: haypo Date: Tue Jul 8 01:21:09 2008 New Revision: 56364 Modified: pypy/dist/pypy/lib/pwd.py Log: Fix getpwnam(): check name type to avoid segfault, eg. getpwnam(False) Modified: pypy/dist/pypy/lib/pwd.py ============================================================================== --- pypy/dist/pypy/lib/pwd.py (original) +++ pypy/dist/pypy/lib/pwd.py Tue Jul 8 01:21:09 2008 @@ -98,6 +98,8 @@ Return the password database entry for the given user name. See pwd.__doc__ for more on password database entries. """ + if not isinstance(name, str): + raise TypeError("expected string") pw = _getpwnam(name) if not pw: raise KeyError("getpwname(): name not found: %s" % name) From haypo at codespeak.net Tue Jul 8 01:26:31 2008 From: haypo at codespeak.net (haypo at codespeak.net) Date: Tue, 8 Jul 2008 01:26:31 +0200 (CEST) Subject: [pypy-svn] r56365 - pypy/dist/pypy/lib Message-ID: <20080707232631.6A51016A13B@codespeak.net> Author: haypo Date: Tue Jul 8 01:26:30 2008 New Revision: 56365 Modified: pypy/dist/pypy/lib/_locale.py Log: Add missing exported symboles in _locale module Modified: pypy/dist/pypy/lib/_locale.py ============================================================================== --- pypy/dist/pypy/lib/_locale.py (original) +++ pypy/dist/pypy/lib/_locale.py Tue Jul 8 01:26:30 2008 @@ -417,7 +417,7 @@ 'setlocale', 'localeconv', 'strxfrm', 'strcoll', 'gettext', 'dgettext', 'dcgettext', 'textdomain', 'bindtextdomain', -) +) + _CONSTANTS if _bind_textdomain_codeset: __all__ += ('bind_textdomain_codeset',) if HAS_LANGINFO: From fijal at codespeak.net Tue Jul 8 07:20:39 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Tue, 8 Jul 2008 07:20:39 +0200 (CEST) Subject: [pypy-svn] r56366 - pypy/extradoc/talk/ep2008 Message-ID: <20080708052039.AA2BA698044@codespeak.net> Author: fijal Date: Tue Jul 8 07:20:37 2008 New Revision: 56366 Added: pypy/extradoc/talk/ep2008/status.pdf (contents, props changed) Log: Add a pdf version of this talk Added: pypy/extradoc/talk/ep2008/status.pdf ============================================================================== Binary file. No diff available. From antocuni at codespeak.net Tue Jul 8 09:20:37 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 8 Jul 2008 09:20:37 +0200 (CEST) Subject: [pypy-svn] r56367 - pypy/extradoc/talk/pycon-italy-2008 Message-ID: <20080708072037.4EEE36980DD@codespeak.net> Author: antocuni Date: Tue Jul 8 09:20:35 2008 New Revision: 56367 Added: pypy/extradoc/talk/pycon-italy-2008/beamerdefs.txt (contents, props changed) Log: forgot to add this file Added: pypy/extradoc/talk/pycon-italy-2008/beamerdefs.txt ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pycon-italy-2008/beamerdefs.txt Tue Jul 8 09:20:35 2008 @@ -0,0 +1,77 @@ +.. colors +.. =========================== + +.. role:: green +.. role:: red + + +.. general useful commands +.. =========================== + +.. |pause| raw:: latex + + \pause + +.. |small| raw:: latex + + {\small + +.. |end_small| raw:: latex + + } + + +.. closed bracket +.. =========================== + +.. |>| raw:: latex + + } + + +.. example block +.. =========================== + +.. |example<| raw:: latex + + \begin{exampleblock}{ + + +.. |end_example| raw:: latex + + \end{exampleblock} + + + +.. alert block +.. =========================== + +.. |alert<| raw:: latex + + \begin{alertblock}{ + + +.. |end_alert| raw:: latex + + \end{alertblock} + + + +.. columns +.. =========================== + +.. |column1| raw:: latex + + \begin{columns} + \begin{column}{0.45\textwidth} + +.. |column2| raw:: latex + + \end{column} + \begin{column}{0.45\textwidth} + + +.. |end_columns| raw:: latex + + \end{column} + \end{columns} From antocuni at codespeak.net Wed Jul 9 09:48:51 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 9 Jul 2008 09:48:51 +0200 (CEST) Subject: [pypy-svn] r56382 - pypy/dist/pypy/translator/goal Message-ID: <20080709074851.E595C169EE7@codespeak.net> Author: antocuni Date: Wed Jul 9 09:48:50 2008 New Revision: 56382 Modified: pypy/dist/pypy/translator/goal/bench-cronjob.py Log: add a version of pypy-jvm with inline threshold=0 to nightly benchmarks Modified: pypy/dist/pypy/translator/goal/bench-cronjob.py ============================================================================== --- pypy/dist/pypy/translator/goal/bench-cronjob.py (original) +++ pypy/dist/pypy/translator/goal/bench-cronjob.py Wed Jul 9 09:48:50 2008 @@ -189,6 +189,7 @@ c--gc=hybrid--_faassen cli--_faassen jvm--_faassen + jvm--inline-threshold=0--_faassen """.split('\n') if backend.strip() and not backend.strip().startswith('#')] print time.ctime() for backend in backends: From antocuni at codespeak.net Wed Jul 9 11:12:48 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 9 Jul 2008 11:12:48 +0200 (CEST) Subject: [pypy-svn] r56383 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080709091248.5EE852A013F@codespeak.net> Author: antocuni Date: Wed Jul 9 11:12:47 2008 New Revision: 56383 Added: pypy/extradoc/sprintinfo/post-ep2008/planning.txt (contents, props changed) Log: add some possible tasks that came to my mind Added: pypy/extradoc/sprintinfo/post-ep2008/planning.txt ============================================================================== --- (empty file) +++ pypy/extradoc/sprintinfo/post-ep2008/planning.txt Wed Jul 9 11:12:47 2008 @@ -0,0 +1,18 @@ +EuroPython sprint +================= + +People Present: + +fill me :-) + +Tasks: + + - try out Python programs and fix them or fix PyPy or fix performance bottlenecks + + - some JIT improvement work + + - port the stackless transform to ootypesystem + + - improve rlib.rjvm (java bindings for rpython) + + - profile & speedup pypy-jvm From fijal at codespeak.net Wed Jul 9 13:39:49 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Wed, 9 Jul 2008 13:39:49 +0200 (CEST) Subject: [pypy-svn] r56385 - pypy/extradoc/talk/ep2008/examples Message-ID: <20080709113949.933B9169E7C@codespeak.net> Author: fijal Date: Wed Jul 9 13:39:47 2008 New Revision: 56385 Added: pypy/extradoc/talk/ep2008/examples/ pypy/extradoc/talk/ep2008/examples/del.py (contents, props changed) pypy/extradoc/talk/ep2008/examples/id.py (contents, props changed) pypy/extradoc/talk/ep2008/examples/module_with_global.py (contents, props changed) pypy/extradoc/talk/ep2008/examples/nobjects.py (contents, props changed) pypy/extradoc/talk/ep2008/examples/resurrections.py (contents, props changed) pypy/extradoc/talk/ep2008/examples/write.py (contents, props changed) Log: Add bunch of examples Added: pypy/extradoc/talk/ep2008/examples/del.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/examples/del.py Wed Jul 9 13:39:47 2008 @@ -0,0 +1,6 @@ +l = [] +class A(object): + def __del__(self): + l.append(3) +A() +print l Added: pypy/extradoc/talk/ep2008/examples/id.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/examples/id.py Wed Jul 9 13:39:47 2008 @@ -0,0 +1,6 @@ + +class A(object): + pass + +x = [A() for i in range(1000000)] +y = [id(i) for i in x] Added: pypy/extradoc/talk/ep2008/examples/module_with_global.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/examples/module_with_global.py Wed Jul 9 13:39:47 2008 @@ -0,0 +1,7 @@ + +class A(object): + def __del__(self): + pass + +a = A() +print a.__del__.func_globals['a'] is a Added: pypy/extradoc/talk/ep2008/examples/nobjects.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/examples/nobjects.py Wed Jul 9 13:39:47 2008 @@ -0,0 +1,18 @@ +from time import clock +d = {} +t0 = clock() + +class A(object): + def __init__(self, previous): + self.previous = previous + +maxi = 100000 +rounds = 10 +for r in range(rounds): + i = 0 + p = A(None) + while i < maxi: + p = A(p) + i += 1 + print clock() - t0 + maxi *= 2 Added: pypy/extradoc/talk/ep2008/examples/resurrections.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/examples/resurrections.py Wed Jul 9 13:39:47 2008 @@ -0,0 +1,14 @@ + +l = [] + +class A(object): + def __del__(self): + print "called" + l.append(self) + +a = A() +del a +import gc +gc.collect() +del l[0] +gc.collect() Added: pypy/extradoc/talk/ep2008/examples/write.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/examples/write.py Wed Jul 9 13:39:47 2008 @@ -0,0 +1,2 @@ +open('xxx', 'w').write('stuff') +assert open('xxx').read() == 'stuff' From haypo at codespeak.net Wed Jul 9 15:42:23 2008 From: haypo at codespeak.net (haypo at codespeak.net) Date: Wed, 9 Jul 2008 15:42:23 +0200 (CEST) Subject: [pypy-svn] r56393 - in pypy/dist/pypy/translator: llvm tool Message-ID: <20080709134223.64B2B2A013F@codespeak.net> Author: haypo Date: Wed Jul 9 15:42:22 2008 New Revision: 56393 Modified: pypy/dist/pypy/translator/llvm/buildllvm.py pypy/dist/pypy/translator/tool/cbuild.py Log: Remove trailing spaces Modified: pypy/dist/pypy/translator/llvm/buildllvm.py ============================================================================== --- pypy/dist/pypy/translator/llvm/buildllvm.py (original) +++ pypy/dist/pypy/translator/llvm/buildllvm.py Wed Jul 9 15:42:22 2008 @@ -12,7 +12,7 @@ def llvm_is_on_path(): if py.path.local.sysfind("llvm-as") is None or \ py.path.local.sysfind("llvm-gcc") is None: - return False + return False return True def exe_version(exe, cache={}): @@ -40,7 +40,7 @@ def have_boehm(): import distutils.sysconfig from os.path import exists - libdir = distutils.sysconfig.EXEC_PREFIX + "/lib" + libdir = distutils.sysconfig.EXEC_PREFIX + "/lib" return exists(libdir + '/libgc.so') or exists(libdir + '/libgc.a') def postfix(): @@ -89,7 +89,7 @@ model = '' if not standalone: model = ' -relocation-model=pic' - + self.cmds.append("llc %s %s.bc -f -o %s.s" % (model, base, base)) self.cmds.append("as %s.s -o %s.o" % (base, base)) @@ -106,7 +106,7 @@ self.cmds.append("gcc %s -c %s -O3 -o %s" % (filename, include_opts, objname)) attrs = self.genllvm.eci._copy_attributes() - attrs['libraries'] = tuple(libraries) + attrs['libraries'] + attrs['libraries'] = tuple(libraries) + attrs['libraries'] self.genllvm.eci = ExternalCompilationInfo(**attrs) # XXX support profile? @@ -134,7 +134,7 @@ self.dirpath.chdir() return self.genllvm.entry_name - + def setup_linker_command(self, base, exename=None): eci = self.genllvm.eci library_files = self.genllvm.db.gcpolicy.gc_libraries() Modified: pypy/dist/pypy/translator/tool/cbuild.py ============================================================================== --- pypy/dist/pypy/translator/tool/cbuild.py (original) +++ pypy/dist/pypy/translator/tool/cbuild.py Wed Jul 9 15:42:22 2008 @@ -450,7 +450,7 @@ class ProfOpt(object): #XXX assuming gcc style flags for now name = "profopt" - + def __init__(self, compiler): self.compiler = compiler @@ -490,7 +490,7 @@ self.include_dirs = list(eci.include_dirs) self.library_dirs = list(eci.library_dirs) self.compile_extra = list(eci.compile_extra) - self.link_extra = list(eci.link_extra) + self.link_extra = list(eci.link_extra) self.frameworks = list(eci.frameworks) self.compiler_exe = compiler_exe self.profbased = profbased @@ -518,7 +518,7 @@ if outputfilename is None: self.outputfilename = py.path.local(cfilenames[0]).new(ext=ext) - else: + else: self.outputfilename = py.path.local(outputfilename) self.eci = eci @@ -565,9 +565,9 @@ if not noerr: print >>sys.stderr, data raise - + def _build(self): - from distutils.ccompiler import new_compiler + from distutils.ccompiler import new_compiler compiler = new_compiler(force=1) if self.compiler_exe is not None: for c in '''compiler compiler_so compiler_cxx @@ -575,7 +575,7 @@ compiler.executables[c][0] = self.compiler_exe compiler.spawn = log_spawned_cmd(compiler.spawn) objects = [] - for cfile in self.cfilenames: + for cfile in self.cfilenames: cfile = py.path.local(cfile) compile_extra = self.compile_extra[:] # -frandom-seed is only to try to be as reproducable as possible @@ -587,17 +587,17 @@ compile_extra = [arg for arg in compile_extra if not arg.startswith('-fprofile-')] - old = cfile.dirpath().chdir() - try: - res = compiler.compile([cfile.basename], + old = cfile.dirpath().chdir() + try: + res = compiler.compile([cfile.basename], include_dirs=self.eci.include_dirs, extra_preargs=compile_extra) assert len(res) == 1 - cobjfile = py.path.local(res[0]) + cobjfile = py.path.local(res[0]) assert cobjfile.check() objects.append(str(cobjfile)) - finally: - old.chdir() + finally: + old.chdir() compiler.link_executable(objects, str(self.outputfilename), libraries=self.eci.libraries, From haypo at codespeak.net Wed Jul 9 15:44:12 2008 From: haypo at codespeak.net (haypo at codespeak.net) Date: Wed, 9 Jul 2008 15:44:12 +0200 (CEST) Subject: [pypy-svn] r56394 - in pypy/dist/pypy/translator: llvm tool Message-ID: <20080709134412.69CF82A013F@codespeak.net> Author: haypo Date: Wed Jul 9 15:44:11 2008 New Revision: 56394 Modified: pypy/dist/pypy/translator/llvm/buildllvm.py pypy/dist/pypy/translator/tool/cbuild.py Log: Support CFLAGS environement variable to cbuild and buildllvm Modified: pypy/dist/pypy/translator/llvm/buildllvm.py ============================================================================== --- pypy/dist/pypy/translator/llvm/buildllvm.py (original) +++ pypy/dist/pypy/translator/llvm/buildllvm.py Wed Jul 9 15:44:11 2008 @@ -15,6 +15,8 @@ return False return True +CFLAGS = os.getenv("CFLAGS") or "-O3" + def exe_version(exe, cache={}): try: v = cache[exe] @@ -84,7 +86,7 @@ use_gcc = self.genllvm.config.translation.llvm_via_c if use_gcc: self.cmds.append("llc %s.bc -march=c -f -o %s.c" % (base, base)) - self.cmds.append("gcc %s.c -c -O3 -fomit-frame-pointer" % base) + self.cmds.append("gcc %s.c -c %s -fomit-frame-pointer" % (base, CFLAGS)) else: model = '' if not standalone: @@ -103,7 +105,7 @@ assert filename.endswith(".c") objname = filename[:-2] + ".o" libraries.add(objname) - self.cmds.append("gcc %s -c %s -O3 -o %s" % (filename, include_opts, objname)) + self.cmds.append("gcc %s -c %s %s -o %s" % (filename, include_opts, CFLAGS, objname)) attrs = self.genllvm.eci._copy_attributes() attrs['libraries'] = tuple(libraries) + attrs['libraries'] @@ -112,13 +114,13 @@ # XXX support profile? # if (self.genllvm.config.translation.profopt is not None and # not self.genllvm.config.translation.noprofopt): -# cmd = "gcc -fprofile-generate %s.c -c -O3 -pipe -o %s.o" % (base, base) +# cmd = "gcc -fprofile-generate %s.c -c %s -pipe -o %s.o" % (base, CFLAGS, base) # self.cmds.append(cmd) # cmd = "gcc -fprofile-generate %s.o %s %s -lm -pipe -o %s_gen" % \ # (base, gc_libs_path, gc_libs, exename) # self.cmds.append(cmd) # self.cmds.append("./%s_gen %s" % (exename, self.genllvm.config.translation.profopt)) -# cmd = "gcc -fprofile-use %s.c -c -O3 -pipe -o %s.o" % (b, b) +# cmd = "gcc -fprofile-use %s.c -c %s -pipe -o %s.o" % (b, CFLAGS, b) # self.cmds.append(cmd) # cmd = "gcc -fprofile-use %s.o %s %s -lm -pipe -o %s" % \ # (b, gc_libs_path, gc_libs, exename) @@ -167,7 +169,7 @@ out = base + ".so" if exename: out = exename - self.cmds.append("gcc -O3 %s.o %s -o %s" % (base, " ".join(compiler_opts), out)) + self.cmds.append("gcc %s %s.o %s -o %s" % (CFLAGS, base, " ".join(compiler_opts), out)) def make_module(self): base = self.setup() Modified: pypy/dist/pypy/translator/tool/cbuild.py ============================================================================== --- pypy/dist/pypy/translator/tool/cbuild.py (original) +++ pypy/dist/pypy/translator/tool/cbuild.py Wed Jul 9 15:44:11 2008 @@ -12,6 +12,12 @@ debug = 0 +CFLAGS = os.getenv("CFLAGS") +if CFLAGS: + CFLAGS = CFLAGS.split() +else: + CFLAGS = ['-O3'] + class ExternalCompilationInfo(object): _ATTRIBUTES = ['pre_include_bits', 'includes', 'include_dirs', @@ -499,7 +505,7 @@ self.libraries.append('m') if 'pthread' not in self.libraries: self.libraries.append('pthread') - self.compile_extra += ['-O3', '-fomit-frame-pointer', '-pthread'] + self.compile_extra += CFLAGS + ['-fomit-frame-pointer', '-pthread'] self.link_extra += ['-pthread'] if sys.platform == 'win32': self.link_extra += ['/DEBUG'] # generate .pdb file @@ -512,7 +518,7 @@ if s + 'lib' not in self.library_dirs and \ os.path.exists(s + 'lib'): self.library_dirs.append(s + 'lib') - self.compile_extra += ['-O3', '-fomit-frame-pointer'] + self.compile_extra += CFLAGS + ['-fomit-frame-pointer'] for framework in self.frameworks: self.link_extra += ['-framework', framework] From arigo at codespeak.net Thu Jul 10 10:17:38 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 10 Jul 2008 10:17:38 +0200 (CEST) Subject: [pypy-svn] r56405 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080710081738.B716B169EFB@codespeak.net> Author: arigo Date: Thu Jul 10 10:17:38 2008 New Revision: 56405 Modified: pypy/extradoc/sprintinfo/post-ep2008/planning.txt Log: Planning for the first day of the sprint. Modified: pypy/extradoc/sprintinfo/post-ep2008/planning.txt ============================================================================== --- pypy/extradoc/sprintinfo/post-ep2008/planning.txt (original) +++ pypy/extradoc/sprintinfo/post-ep2008/planning.txt Thu Jul 10 10:17:38 2008 @@ -3,16 +3,34 @@ People Present: -fill me :-) +jakub, gasper, david, andrew, jonathan, geoff, anto, stephan, +henrik, dinu, jacob, marius, samuele, fijal, arigo Tasks: - - try out Python programs and fix them or fix PyPy or fix performance bottlenecks - - - some JIT improvement work + - try out Python programs and fix them or fix PyPy or fix + performance bottlenecks (geoff, henrik, david, fijal around) + + - support for profiling the calls to builtins (stephan, fijal + around) + + - fix the default options of "translate.py", make "--help" + show only the reasonable options, remove "--faassen", + etc. (samuele, others for discussion) + + - dotviewer improvements (expanding & collapsing, etc.) + and eventually make it a separate project (dinu, marius) + - start a tutorial on how to write a language interpreter + in RPython with PyPy (jonathan, ...) + - port the stackless transform to ootypesystem - improve rlib.rjvm (java bindings for rpython) - profile & speedup pypy-jvm + + - more extension modules, with ctypes or rpython (andrew, + jakub, gasper, ...) + + - general wizardry (anto, arigo) From fijal at codespeak.net Thu Jul 10 10:42:47 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Thu, 10 Jul 2008 10:42:47 +0200 (CEST) Subject: [pypy-svn] r56406 - pypy/dist/pypy/interpreter/test Message-ID: <20080710084247.1CC2D169F54@codespeak.net> Author: fijal Date: Thu Jul 10 10:42:45 2008 New Revision: 56406 Modified: pypy/dist/pypy/interpreter/test/test_executioncontext.py Log: (fijal, stephan) A test for builtin-level profiling Modified: pypy/dist/pypy/interpreter/test/test_executioncontext.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_executioncontext.py (original) +++ pypy/dist/pypy/interpreter/test/test_executioncontext.py Thu Jul 10 10:42:45 2008 @@ -62,17 +62,19 @@ assert space.sys.checkinterval / 10 < i < space.sys.checkinterval * 3 def test_llprofile(self): + py.test.skip("not working yet") l = [] - def profile_func(space, w_arg, event, frame, w_aarg): + def profile_func(space, w_arg, frame, event, w_aarg): assert w_arg is space.w_None - l.append(frame) + l.append(event) space = self.space space.getexecutioncontext().setllprofile(profile_func, space.w_None) space.appexec([], """(): - pass + l = [] + l.append(3) """) space.getexecutioncontext().setllprofile(None, None) - assert l == ['call', 'return', 'call', 'return'] + assert l == ['call', 'return', 'call', 'c_call', 'c_return', 'return'] From jlg at codespeak.net Thu Jul 10 12:33:51 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Thu, 10 Jul 2008 12:33:51 +0200 (CEST) Subject: [pypy-svn] r56407 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710103351.1C3C316A0C2@codespeak.net> Author: jlg Date: Thu Jul 10 12:33:49 2008 New Revision: 56407 Added: pypy/dist/pypy/module/itertools/ (props changed) pypy/dist/pypy/module/itertools/__init__.py (contents, props changed) pypy/dist/pypy/module/itertools/interp_itertools.py (contents, props changed) pypy/dist/pypy/module/itertools/test/ (props changed) pypy/dist/pypy/module/itertools/test/test_itertools.py (contents, props changed) Log: (andrew, jlg) started intepreter level itertools module; count implemented Added: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/module/itertools/__init__.py Thu Jul 10 12:33:49 2008 @@ -0,0 +1,12 @@ + +from pypy.interpreter.mixedmodule import MixedModule + +class Module(MixedModule): + """An itertools module.""" + + interpleveldefs = { + 'count' : 'interp_itertools.W_Count', + } + + appleveldefs = { + } Added: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 12:33:49 2008 @@ -0,0 +1,62 @@ +from pypy.interpreter.baseobjspace import Wrappable +from pypy.interpreter.error import OperationError +from pypy.interpreter.typedef import TypeDef +from pypy.interpreter.gateway import interp2app, ObjSpace, W_Root +from pypy.rlib.rarithmetic import ovfcheck + +class W_Count(Wrappable): + + def __init__(self, space, firstval): + self.space = space + self.c = firstval + self.overflowed = False + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if self.overflowed: + raise OperationError(self.space.w_OverflowError, + self.space.wrap("cannot count beyond sys.maxint")) + + c = self.c + try: + self.c = ovfcheck(self.c + 1) + except OverflowError: + self.overflowed = True + + return self.space.wrap(c) + + +def W_Count___new__(space, w_subtype, firstval=0): + """ + Create a new count object and call its initializer. + """ + return space.wrap(W_Count(space, firstval)) + +W_Count.typedef = TypeDef( + 'count', + __new__ = interp2app(W_Count___new__, unwrap_spec=[ObjSpace, W_Root, int]), + __iter__ = interp2app(W_Count.iter_w, unwrap_spec=['self']), + next = interp2app(W_Count.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that returns consecutive integers starting + with n. If not specified n defaults to zero. Does not currently + support python long integers. Often used as an argument to imap() + to generate consecutive data points. Also, used with izip() to + add sequence numbers. + + Equivalent to : + + def count(n=0): + if not isinstance(n, int): + raise TypeError("%s is not a regular integer" % n) + while True: + yield n + n += 1 + """) + +class W_Repeat(Wrappable): + + def __init__(self, space): + self.space = space + Added: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 12:33:49 2008 @@ -0,0 +1,47 @@ +from pypy.conftest import gettestobjspace + +class AppTestItertools: + + def setup_class(cls): + cls.space = gettestobjspace(usemodules=['itertools']) + + def test_count(self): + import itertools + + c = itertools.count() + for x in range(10): + assert c.next() == x + + def test_count_iterable(self): + import itertools + + c = itertools.count() + assert hasattr(c, '__iter__') + assert iter(c) is c + assert hasattr(c, 'next') + + def test_count_param(self): + import itertools + + c = itertools.count(3) + for x in range(10): + assert c.next() == x + 3 + + def test_count_overflow(self): + import itertools, sys + + c = itertools.count(sys.maxint) + assert c.next() == sys.maxint + raises(OverflowError, c.next) + raises(OverflowError, c.next) + + def test_repeat(self): + skip("for now") + import itertools + + o = object() + r = itertools.repeat(o) + + for x in range(10): + assert o is r.next() + From fijal at codespeak.net Thu Jul 10 14:26:48 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Thu, 10 Jul 2008 14:26:48 +0200 (CEST) Subject: [pypy-svn] r56411 - pypy/dist/pypy/rlib Message-ID: <20080710122648.877EE2A00DC@codespeak.net> Author: fijal Date: Thu Jul 10 14:26:46 2008 New Revision: 56411 Modified: pypy/dist/pypy/rlib/libffi.py Log: Fix c library finding. Modified: pypy/dist/pypy/rlib/libffi.py ============================================================================== --- pypy/dist/pypy/rlib/libffi.py (original) +++ pypy/dist/pypy/rlib/libffi.py Thu Jul 10 14:26:46 2008 @@ -11,7 +11,7 @@ from pypy.translator.tool.cbuild import ExternalCompilationInfo import py import os - +import ctypes.util DEBUG = False # writes dlerror() messages to stderr @@ -170,8 +170,7 @@ return rffi.llexternal(name, args, result, compilation_info=eci, calling_conv='win') if not _MS_WINDOWS: - c_dlopen = external('dlopen', [rffi.CCHARP, rffi.INT], rffi.VOIDP, - _nowrapper=True) + c_dlopen = external('dlopen', [rffi.CCHARP, rffi.INT], rffi.VOIDP) c_dlclose = external('dlclose', [rffi.VOIDP], rffi.INT) c_dlerror = external('dlerror', [], rffi.CCHARP) c_dlsym = external('dlsym', [rffi.VOIDP, rffi.CCHARP], rffi.VOIDP) @@ -220,7 +219,7 @@ return res def get_libc_name(): - return 'libc.so.6' + return ctypes.util.find_library('c') if _MS_WINDOWS: def dlopen(name): From henrikv at codespeak.net Thu Jul 10 14:57:33 2008 From: henrikv at codespeak.net (henrikv at codespeak.net) Date: Thu, 10 Jul 2008 14:57:33 +0200 (CEST) Subject: [pypy-svn] r56412 - in pypy/dist/pypy: module/_rawffi/test rlib rlib/test Message-ID: <20080710125733.E9DAE2A0152@codespeak.net> Author: henrikv Date: Thu Jul 10 14:57:32 2008 New Revision: 56412 Modified: pypy/dist/pypy/module/_rawffi/test/test__rawffi.py pypy/dist/pypy/rlib/libffi.py pypy/dist/pypy/rlib/test/test_libffi.py Log: (henrikv, fijal) Make libffi work on MACOSX Modified: pypy/dist/pypy/module/_rawffi/test/test__rawffi.py ============================================================================== --- pypy/dist/pypy/module/_rawffi/test/test__rawffi.py (original) +++ pypy/dist/pypy/module/_rawffi/test/test__rawffi.py Thu Jul 10 14:57:32 2008 @@ -8,10 +8,6 @@ import os, sys, py -def setup_module(mod): - if sys.platform not in ('linux2', 'win32'): - py.test.skip("Linux & win32 only tests by now") - class AppTestFfi: def prepare_c_example(): from pypy.tool.udir import udir @@ -162,17 +158,20 @@ prepare_c_example = staticmethod(prepare_c_example) def setup_class(cls): + from pypy.rlib.libffi import get_libc_name space = gettestobjspace(usemodules=('_rawffi','struct')) cls.space = space cls.w_lib_name = space.wrap(cls.prepare_c_example()) + cls.w_libc_name = space.wrap(get_libc_name()) if sys.platform == 'win32': cls.w_iswin32 = space.wrap(True) - cls.w_libc_name = space.wrap('msvcrt') cls.w_libm_name = space.wrap('msvcrt') else: cls.w_iswin32 = space.wrap(False) - cls.w_libc_name = space.wrap('libc.so.6') cls.w_libm_name = space.wrap('libm.so') + if sys.platform == "darwin": + cls.w_libm_name = space.wrap('libm.dylib') + cls.w_sizes_and_alignments = space.wrap(dict( [(k, (v.c_size, v.c_alignment)) for k,v in TYPEMAP.iteritems()])) Modified: pypy/dist/pypy/rlib/libffi.py ============================================================================== --- pypy/dist/pypy/rlib/libffi.py (original) +++ pypy/dist/pypy/rlib/libffi.py Thu Jul 10 14:57:32 2008 @@ -11,11 +11,13 @@ from pypy.translator.tool.cbuild import ExternalCompilationInfo import py import os +import sys import ctypes.util DEBUG = False # writes dlerror() messages to stderr _MS_WINDOWS = os.name == "nt" +_MAC_OS = sys.platform == "darwin" if _MS_WINDOWS: from pypy.rlib import rwin32 @@ -23,7 +25,10 @@ if not _MS_WINDOWS: includes = ['dlfcn.h', 'ffi.h'] include_dirs = [] - pot_incl = py.path.local('/usr/include/libffi') + if _MAC_OS: + pot_incl = py.path.local('/usr/include/ffi') + else: + pot_incl = py.path.local('/usr/include/libffi') if pot_incl.check(): include_dirs.append(str(pot_incl)) lib_dirs = [] @@ -31,7 +36,12 @@ if pot_lib.check(): lib_dirs.append(str(pot_lib)) + if _MAC_OS: + pre_include_bits = ['#define MACOSX'] + else: + pre_include_bits = [] eci = ExternalCompilationInfo( + pre_include_bits = pre_include_bits, includes = includes, libraries = ['ffi', 'dl'], include_dirs = include_dirs, Modified: pypy/dist/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/dist/pypy/rlib/test/test_libffi.py (original) +++ pypy/dist/pypy/rlib/test/test_libffi.py Thu Jul 10 14:57:32 2008 @@ -13,9 +13,6 @@ import time def setup_module(mod): - if not (sys.platform.startswith('linux') or - sys.platform == 'win32'): - py.test.skip("Fragile tests, linux & win32 only by now") for name in type_names: # XXX force this to be seen by ll2ctypes # so that ALLOCATED.clear() clears it @@ -28,20 +25,16 @@ def test_dlopen(self): py.test.raises(OSError, "dlopen(rffi.str2charp('xxxxxxxxxxxx'))") - if sys.platform == 'win32': - assert dlopen(rffi.str2charp('kernel32.dll')) - else: - assert dlopen(rffi.str2charp('/lib/libc.so.6')) + assert dlopen(rffi.str2charp(get_libc_name())) def get_libc(self): - if sys.platform == 'win32': - return CDLL('msvcrt.dll') - else: - return CDLL('/lib/libc.so.6') + return CDLL(get_libc_name()) def get_libm(self): if sys.platform == 'win32': return CDLL('msvcrt.dll') + elif sys.platform == "darwin": + return CDLL('libm.dylib') else: return CDLL('libm.so') From adurdin at codespeak.net Thu Jul 10 15:19:34 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Thu, 10 Jul 2008 15:19:34 +0200 (CEST) Subject: [pypy-svn] r56413 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710131934.55D15169F2F@codespeak.net> Author: adurdin Date: Thu Jul 10 15:19:34 2008 New Revision: 56413 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) implemented repeat() in interp_itertools. Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Thu Jul 10 15:19:34 2008 @@ -6,6 +6,7 @@ interpleveldefs = { 'count' : 'interp_itertools.W_Count', + 'repeat' : 'interp_itertools.W_Repeat', } appleveldefs = { Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 15:19:34 2008 @@ -55,8 +55,54 @@ n += 1 """) + class W_Repeat(Wrappable): - def __init__(self, space): + def __init__(self, space, w_obj, w_times): self.space = space + self.w_obj = w_obj + + if space.is_w(w_times, space.w_None): + self.counting = False + self.count = 0 + else: + self.counting = True + self.count = self.space.int_w(w_times) + + def next_w(self): + if self.counting: + if self.count <= 0: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + self.count -= 1 + return self.w_obj + + def iter_w(self): + return self.space.wrap(self) + +def W_Repeat___new__(space, w_subtype, w_obj, w_times=None): + """ + Create a new repeat object and call its initializer. + """ + return space.wrap(W_Repeat(space, w_obj, w_times)) +W_Repeat.typedef = TypeDef( + 'repeat', + __new__ = interp2app(W_Repeat___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), + __iter__ = interp2app(W_Repeat.iter_w, unwrap_spec=['self']), + next = interp2app(W_Repeat.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that returns object over and over again. + Runs indefinitely unless the times argument is specified. Used + as argument to imap() for invariant parameters to the called + function. Also used with izip() to create an invariant part of a + tuple record. + + Equivalent to : + + def repeat(object, times=None): + if times is None: + while True: + yield object + else: + for i in xrange(times): + yield object + """) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 15:19:34 2008 @@ -1,26 +1,31 @@ from pypy.conftest import gettestobjspace class AppTestItertools: - def setup_class(cls): cls.space = gettestobjspace(usemodules=['itertools']) - def test_count(self): + def test_iterables(self): import itertools - c = itertools.count() - for x in range(10): - assert c.next() == x + iterables = [ + itertools.count(), + itertools.repeat(None), + ] + + for it in iterables: + assert hasattr(it, '__iter__') + assert iter(it) is it + assert hasattr(it, 'next') + assert callable(it.next) - def test_count_iterable(self): + def test_count(self): import itertools c = itertools.count() - assert hasattr(c, '__iter__') - assert iter(c) is c - assert hasattr(c, 'next') + for x in range(10): + assert c.next() == x - def test_count_param(self): + def test_count_firstval(self): import itertools c = itertools.count(3) @@ -35,8 +40,9 @@ raises(OverflowError, c.next) raises(OverflowError, c.next) + raises(OverflowError, itertools.count, sys.maxint + 1) + def test_repeat(self): - skip("for now") import itertools o = object() @@ -45,3 +51,29 @@ for x in range(10): assert o is r.next() + def test_repeat_times(self): + import itertools + + times = 10 + r = itertools.repeat(None, times=times) + for i in range(times): + r.next() + raises(StopIteration, r.next) + + r = itertools.repeat(None, times=None) + for x in range(10): + r.next() # Should be no StopIteration + + r = itertools.repeat(None, times=0) + raises(StopIteration, r.next) + raises(StopIteration, r.next) + + r = itertools.repeat(None, times=-1) + raises(StopIteration, r.next) + raises(StopIteration, r.next) + + def test_repeat_overflow(self): + import itertools + import sys + + raises(OverflowError, itertools.repeat, None, sys.maxint + 1) From antocuni at codespeak.net Thu Jul 10 15:21:36 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 10 Jul 2008 15:21:36 +0200 (CEST) Subject: [pypy-svn] r56414 - in pypy/branch/oo-jit/pypy/jit/rainbow: . test Message-ID: <20080710132136.88B54169F7B@codespeak.net> Author: antocuni Date: Thu Jul 10 15:21:33 2008 New Revision: 56414 Modified: pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py pypy/branch/oo-jit/pypy/jit/rainbow/interpreter.py pypy/branch/oo-jit/pypy/jit/rainbow/test/test_interpreter.py pypy/branch/oo-jit/pypy/jit/rainbow/test/test_promotion.py Log: make sure to remember x.__class__ after a promote_class. Right now it's a complete hack on ootype, but since these things will change as soon as we merge the less-meta-instances branch, it's not worth to spend more time on it. Modified: pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py (original) +++ pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py Thu Jul 10 15:21:33 2008 @@ -1001,10 +1001,17 @@ check=False) fielddescindex = self.fielddesc_position(self.OBJECT, self.classfieldname) + # hack hack hack, which will go away when we merge less-meta-instances + if self.__class__.__name__ == 'OOTypeBytecodeWriter': + fielddescindex2 = self.fielddesc_position(v_obj.concretetype, 'meta') + else: + fielddescindex2 = fielddescindex + self.emit("assert_class") self.emit(self.serialize_oparg("red", v_obj)) self.emit(g_class) self.emit(fielddescindex) + self.emit(fielddescindex2) self.register_redvar(result) Modified: pypy/branch/oo-jit/pypy/jit/rainbow/interpreter.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/rainbow/interpreter.py (original) +++ pypy/branch/oo-jit/pypy/jit/rainbow/interpreter.py Thu Jul 10 15:21:33 2008 @@ -629,12 +629,16 @@ assert gv_switchvar.is_const self.green_result(gv_switchvar) - @arguments("red", "green", "fielddesc", returns="red") - def opimpl_assert_class(self, objbox, gv_class, fielddesc): + @arguments("red", "green", "fielddesc", "fielddesc", returns="red") + def opimpl_assert_class(self, objbox, gv_class, fielddesc, fielddesc2): if isinstance(objbox.content, rcontainer.VirtualStruct): return objbox classbox = self.PtrRedBox(gv_class) objbox.remember_field(fielddesc, classbox) + + # hack hack, see comments in codewriter.py + if fielddesc2 != fielddesc: + objbox.remember_field(fielddesc2, classbox) return objbox @arguments() Modified: pypy/branch/oo-jit/pypy/jit/rainbow/test/test_interpreter.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/rainbow/test/test_interpreter.py (original) +++ pypy/branch/oo-jit/pypy/jit/rainbow/test/test_interpreter.py Thu Jul 10 15:21:33 2008 @@ -300,11 +300,12 @@ assert oops.get(name, 0) == count def check_flexswitches(self, expected_count): + from pypy.rpython.ootypesystem import rclass residual_graph = self.get_residual_graph() count = 0 for block in residual_graph.iterblocks(): if (isinstance(block.exitswitch, Variable) and - block.exitswitch.concretetype is lltype.Signed): + block.exitswitch.concretetype in (lltype.Signed, rclass.CLASSTYPE)): count += 1 assert count == expected_count @@ -2307,6 +2308,7 @@ replace = { 'getfield': 'oogetfield', 'setfield': 'oosetfield', + 'malloc': 'new', } insns = insns.copy() Modified: pypy/branch/oo-jit/pypy/jit/rainbow/test/test_promotion.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/rainbow/test/test_promotion.py (original) +++ pypy/branch/oo-jit/pypy/jit/rainbow/test/test_promotion.py Thu Jul 10 15:21:33 2008 @@ -497,7 +497,7 @@ res = self.interpret(ll_function, [20, True], [], policy=StopAtXPolicy(make_obj)) assert res == 42 self.check_insns(malloc=0) - self.check_insns(new=0) + self.check_flexswitches(3) def test_promote_class_vstruct(self): class A: @@ -511,7 +511,25 @@ res = self.interpret(ll_function, [], []) assert res == 42 self.check_insns(malloc=0) - self.check_insns(new=0) + + def test_read___class___after_promotion(self): + class A: + pass + class B(A): + pass + + def make_obj(flag): + return flag and A() or B() + + def ll_function(flag): + hint(None, global_merge_point=True) + obj = make_obj(flag) + promoted_obj = hint(obj, promote_class=True) + cls = promoted_obj.__class__ + return cls is A + + res = self.interpret(ll_function, [True], [], policy=StopAtXPolicy(make_obj)) + self.check_flexswitches(2) class TestLLType(BaseTestPromotion): type_system = "lltype" From antocuni at codespeak.net Thu Jul 10 15:27:10 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 10 Jul 2008 15:27:10 +0200 (CEST) Subject: [pypy-svn] r56415 - in pypy/branch/oo-jit/pypy: jit/rainbow/test jit/tl rpython/ootypesystem Message-ID: <20080710132710.A1FCD169F7F@codespeak.net> Author: antocuni Date: Thu Jul 10 15:27:10 2008 New Revision: 56415 Modified: pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py pypy/branch/oo-jit/pypy/jit/tl/tlc.py pypy/branch/oo-jit/pypy/rpython/ootypesystem/rclass.py Log: cool, tlc starts working properly on ootype :-) Modified: pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py (original) +++ pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py Thu Jul 10 15:27:10 2008 @@ -2,6 +2,7 @@ from pypy.rpython.module.support import LLSupport, OOSupport from pypy.jit.rainbow.test.test_portal import PortalTest from pypy.jit.rainbow.test.test_vlist import P_OOPSPEC +from pypy.jit.rainbow.test.test_interpreter import OOTypeMixin from pypy.tool.sourcetools import func_with_new_name from pypy.jit.conftest import Benchmark @@ -69,6 +70,9 @@ type_system = "lltype" to_rstr = staticmethod(LLSupport.to_rstr) -##class TestOOType(BaseTestTLC): -## type_system = "ootype" -## to_rstr = staticmethod(OOSupport.to_rstr) +class TestOOType(OOTypeMixin, BaseTestTLC): + type_system = "ootype" + to_rstr = staticmethod(OOSupport.to_rstr) + + def test_nth_item(self): + py.test.skip('in progress') Modified: pypy/branch/oo-jit/pypy/jit/tl/tlc.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/tl/tlc.py (original) +++ pypy/branch/oo-jit/pypy/jit/tl/tlc.py Thu Jul 10 15:27:10 2008 @@ -180,62 +180,62 @@ elif opcode == ADD: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(b.add(a)) elif opcode == SUB: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(b.sub(a)) elif opcode == MUL: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(b.mul(a)) elif opcode == DIV: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(b.div(a)) elif opcode == EQ: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(IntObj(b.eq(a))) elif opcode == NE: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(IntObj(not b.eq(a))) elif opcode == LT: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(IntObj(b.lt(a))) elif opcode == LE: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(IntObj(not a.lt(b))) elif opcode == GT: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(IntObj(a.lt(b))) elif opcode == GE: a, b = stack.pop(), stack.pop() - hint(a.__class__, promote=True) - hint(b.__class__, promote=True) + hint(a, promote_class=True) + hint(b, promote_class=True) stack.append(IntObj(not b.lt(a))) elif opcode == BR_COND: Modified: pypy/branch/oo-jit/pypy/rpython/ootypesystem/rclass.py ============================================================================== --- pypy/branch/oo-jit/pypy/rpython/ootypesystem/rclass.py (original) +++ pypy/branch/oo-jit/pypy/rpython/ootypesystem/rclass.py Thu Jul 10 15:27:10 2008 @@ -549,4 +549,5 @@ # type(None) -> NULL (for now) return ootype.null(CLASSTYPE) -ll_type = ll_inst_type +def ll_type(obj): + return obj.meta From antocuni at codespeak.net Thu Jul 10 15:57:04 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 10 Jul 2008 15:57:04 +0200 (CEST) Subject: [pypy-svn] r56416 - in pypy/branch/oo-jit/pypy: jit/rainbow jit/rainbow/test rpython/ootypesystem Message-ID: <20080710135704.3D335169FF0@codespeak.net> Author: antocuni Date: Thu Jul 10 15:57:02 2008 New Revision: 56416 Modified: pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py pypy/branch/oo-jit/pypy/rpython/ootypesystem/ootype.py Log: make sure to check all subclasses when computing methodcodes. This fixes test_0tlc. Modified: pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py (original) +++ pypy/branch/oo-jit/pypy/jit/rainbow/codewriter.py Thu Jul 10 15:57:02 2008 @@ -1908,7 +1908,7 @@ def fill_methodcodes(self, INSTANCE, methname, graph2tsgraph): class2typedesc = self.interpreter.class2typedesc - TYPES = [INSTANCE] + INSTANCE._subclasses + TYPES = INSTANCE._all_subclasses() for T in TYPES: descindex = self.structtypedesc_position(T) desc = self.structtypedescs[descindex] Modified: pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py (original) +++ pypy/branch/oo-jit/pypy/jit/rainbow/test/test_0tlc.py Thu Jul 10 15:57:02 2008 @@ -73,6 +73,3 @@ class TestOOType(OOTypeMixin, BaseTestTLC): type_system = "ootype" to_rstr = staticmethod(OOSupport.to_rstr) - - def test_nth_item(self): - py.test.skip('in progress') Modified: pypy/branch/oo-jit/pypy/rpython/ootypesystem/ootype.py ============================================================================== --- pypy/branch/oo-jit/pypy/rpython/ootypesystem/ootype.py (original) +++ pypy/branch/oo-jit/pypy/rpython/ootypesystem/ootype.py Thu Jul 10 15:57:02 2008 @@ -118,6 +118,21 @@ assert isinstance(INSTANCE, Instance) self._subclasses.append(INSTANCE) + def _all_subclasses(self): + """ + Transitive closure on self._subclasses. + + Return a set containing all direct and indirect subclasses, + including itself. + """ + res = set() + stack = [self] + while stack: + item = stack.pop() + res.add(item) + stack += item._subclasses + return res + def _add_fields(self, fields): fields = fields.copy() # mutated below for name, defn in fields.iteritems(): From jlg at codespeak.net Thu Jul 10 16:11:51 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Thu, 10 Jul 2008 16:11:51 +0200 (CEST) Subject: [pypy-svn] r56417 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710141151.E600D16A061@codespeak.net> Author: jlg Date: Thu Jul 10 16:11:51 2008 New Revision: 56417 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - takewhile implemented Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Thu Jul 10 16:11:51 2008 @@ -5,8 +5,9 @@ """An itertools module.""" interpleveldefs = { - 'count' : 'interp_itertools.W_Count', + 'count' : 'interp_itertools.W_Count', 'repeat' : 'interp_itertools.W_Repeat', + 'takewhile' : 'interp_itertools.W_TakeWhile', } appleveldefs = { Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 16:11:51 2008 @@ -106,3 +106,52 @@ for i in xrange(times): yield object """) + +class W_TakeWhile(Wrappable): + + def __init__(self, space, w_predicate, w_iterable): + self.space = space + self.w_predicate = w_predicate + self.iterable = space.iter(w_iterable) + self.stopped = False + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if self.stopped: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + try: + w_obj = self.space.next(self.iterable) + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + w_bool = self.space.call_function(self.w_predicate, w_obj) + if not self.space.is_true(w_bool): + self.stopped = True + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + return w_obj + +def W_TakeWhile___new__(space, w_subtype, w_predicate, w_iterable): + return space.wrap(W_TakeWhile(space, w_predicate, w_iterable)) + + +W_TakeWhile.typedef = TypeDef( + 'takewhile', + __new__ = interp2app(W_TakeWhile___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), + __iter__ = interp2app(W_TakeWhile.iter_w, unwrap_spec=['self']), + next = interp2app(W_TakeWhile.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that returns elements from the iterable as + long as the predicate is true. + + Equivalent to : + + def takewhile(predicate, iterable): + for x in iterable: + if predicate(x): + yield x + else: + break + """) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 16:11:51 2008 @@ -10,6 +10,7 @@ iterables = [ itertools.count(), itertools.repeat(None), + itertools.takewhile(bool, []), ] for it in iterables: @@ -77,3 +78,26 @@ import sys raises(OverflowError, itertools.repeat, None, sys.maxint + 1) + + def test_takewhile(self): + import itertools + + tw = itertools.takewhile(bool, []) + raises(StopIteration, tw.next) + + tw = itertools.takewhile(bool, [False, True, True]) + raises(StopIteration, tw.next) + + tw = itertools.takewhile(bool, [1, 2, 3, 0, 1, 1]) + for x in range(3): + assert tw.next() == x + 1 + + raises(StopIteration, tw.next) + + def test_takewhile_wrongargs(self): + import itertools + + tw = itertools.takewhile(None, [1]) + raises(TypeError, tw.next) + + raises(TypeError, itertools.takewhile, bool, None) From adurdin at codespeak.net Thu Jul 10 16:29:19 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Thu, 10 Jul 2008 16:29:19 +0200 (CEST) Subject: [pypy-svn] r56418 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710142919.DDD0416A0A3@codespeak.net> Author: adurdin Date: Thu Jul 10 16:29:18 2008 New Revision: 56418 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) inter_itertools - implemented dropwhile Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Thu Jul 10 16:29:18 2008 @@ -6,6 +6,7 @@ interpleveldefs = { 'count' : 'interp_itertools.W_Count', + 'dropwhile' : 'interp_itertools.W_DropWhile', 'repeat' : 'interp_itertools.W_Repeat', 'takewhile' : 'interp_itertools.W_TakeWhile', } Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 16:29:18 2008 @@ -155,3 +155,60 @@ else: break """) + +class W_DropWhile(Wrappable): + + def __init__(self, space, w_predicate, w_iterable): + self.space = space + self.w_predicate = w_predicate + self.iterable = space.iter(w_iterable) + self.started = False + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if self.started: + try: + w_obj = self.space.next(self.iterable) + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + else: + while True: + try: + w_obj = self.space.next(self.iterable) + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + w_bool = self.space.call_function(self.w_predicate, w_obj) + if not self.space.is_true(w_bool): + self.started = True + break + + return w_obj + +def W_DropWhile___new__(space, w_subtype, w_predicate, w_iterable): + return space.wrap(W_DropWhile(space, w_predicate, w_iterable)) + + +W_DropWhile.typedef = TypeDef( + 'dropwhile', + __new__ = interp2app(W_DropWhile___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), + __iter__ = interp2app(W_DropWhile.iter_w, unwrap_spec=['self']), + next = interp2app(W_DropWhile.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that drops elements from the iterable as long + as the predicate is true; afterwards, returns every + element. Note, the iterator does not produce any output until the + predicate is true, so it may have a lengthy start-up time. + + Equivalent to : + + def dropwhile(predicate, iterable): + iterable = iter(iterable) + for x in iterable: + if not predicate(x): + yield x + break + for x in iterable: + yield x + """) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 16:29:18 2008 @@ -11,6 +11,7 @@ itertools.count(), itertools.repeat(None), itertools.takewhile(bool, []), + itertools.dropwhile(bool, []), ] for it in iterables: @@ -22,24 +23,24 @@ def test_count(self): import itertools - c = itertools.count() + it = itertools.count() for x in range(10): - assert c.next() == x + assert it.next() == x def test_count_firstval(self): import itertools - c = itertools.count(3) + it = itertools.count(3) for x in range(10): - assert c.next() == x + 3 + assert it.next() == x + 3 def test_count_overflow(self): import itertools, sys - c = itertools.count(sys.maxint) - assert c.next() == sys.maxint - raises(OverflowError, c.next) - raises(OverflowError, c.next) + it = itertools.count(sys.maxint) + assert it.next() == sys.maxint + raises(OverflowError, it.next) + raises(OverflowError, it.next) raises(OverflowError, itertools.count, sys.maxint + 1) @@ -47,31 +48,31 @@ import itertools o = object() - r = itertools.repeat(o) + it = itertools.repeat(o) for x in range(10): - assert o is r.next() + assert o is it.next() def test_repeat_times(self): import itertools times = 10 - r = itertools.repeat(None, times=times) + it = itertools.repeat(None, times=times) for i in range(times): - r.next() - raises(StopIteration, r.next) + it.next() + raises(StopIteration, it.next) - r = itertools.repeat(None, times=None) + it = itertools.repeat(None, times=None) for x in range(10): - r.next() # Should be no StopIteration + it.next() # Should be no StopIteration - r = itertools.repeat(None, times=0) - raises(StopIteration, r.next) - raises(StopIteration, r.next) - - r = itertools.repeat(None, times=-1) - raises(StopIteration, r.next) - raises(StopIteration, r.next) + it = itertools.repeat(None, times=0) + raises(StopIteration, it.next) + raises(StopIteration, it.next) + + it = itertools.repeat(None, times=-1) + raises(StopIteration, it.next) + raises(StopIteration, it.next) def test_repeat_overflow(self): import itertools @@ -82,22 +83,48 @@ def test_takewhile(self): import itertools - tw = itertools.takewhile(bool, []) - raises(StopIteration, tw.next) + it = itertools.takewhile(bool, []) + raises(StopIteration, it.next) - tw = itertools.takewhile(bool, [False, True, True]) - raises(StopIteration, tw.next) + it = itertools.takewhile(bool, [False, True, True]) + raises(StopIteration, it.next) - tw = itertools.takewhile(bool, [1, 2, 3, 0, 1, 1]) - for x in range(3): - assert tw.next() == x + 1 + it = itertools.takewhile(bool, [1, 2, 3, 0, 1, 1]) + for x in [1, 2, 3]: + assert it.next() == x - raises(StopIteration, tw.next) + raises(StopIteration, it.next) def test_takewhile_wrongargs(self): import itertools - tw = itertools.takewhile(None, [1]) - raises(TypeError, tw.next) + it = itertools.takewhile(None, [1]) + raises(TypeError, it.next) raises(TypeError, itertools.takewhile, bool, None) + + def test_dropwhile(self): + import itertools + + it = itertools.dropwhile(bool, []) + raises(StopIteration, it.next) + + it = itertools.dropwhile(bool, [True, True, True]) + raises(StopIteration, it.next) + + def is_odd(arg): + return (arg % 2 == 1) + + it = itertools.dropwhile(is_odd, [1, 3, 5, 2, 4, 6]) + for x in [2, 4, 6]: + assert it.next() == x + + raises(StopIteration, it.next) + + def test_takewhile_wrongargs(self): + import itertools + + it = itertools.dropwhile(None, [1]) + raises(TypeError, it.next) + + raises(TypeError, itertools.dropwhile, bool, None) From adurdin at codespeak.net Thu Jul 10 16:47:28 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Thu, 10 Jul 2008 16:47:28 +0200 (CEST) Subject: [pypy-svn] r56419 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710144728.8AD112A00DC@codespeak.net> Author: adurdin Date: Thu Jul 10 16:47:27 2008 New Revision: 56419 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - implemented ifilter Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Thu Jul 10 16:47:27 2008 @@ -7,6 +7,7 @@ interpleveldefs = { 'count' : 'interp_itertools.W_Count', 'dropwhile' : 'interp_itertools.W_DropWhile', + 'ifilter' : 'interp_itertools.W_IFilter', 'repeat' : 'interp_itertools.W_Repeat', 'takewhile' : 'interp_itertools.W_TakeWhile', } Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 16:47:27 2008 @@ -196,7 +196,7 @@ __new__ = interp2app(W_DropWhile___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), __iter__ = interp2app(W_DropWhile.iter_w, unwrap_spec=['self']), next = interp2app(W_DropWhile.next_w, unwrap_spec=['self']), - __doc__ = """Make an iterator that drops elements from the iterable as long + __doc__ = """Make an iterator that drops elements from the iterable as long as the predicate is true; afterwards, returns every element. Note, the iterator does not produce any output until the predicate is true, so it may have a lengthy start-up time. @@ -212,3 +212,50 @@ for x in iterable: yield x """) + + +class W_IFilter(Wrappable): + + def __init__(self, space, w_predicate, w_iterable): + self.space = space + if space.is_w(w_predicate, space.w_None): + self.w_predicate = space.w_bool + else: + self.w_predicate = w_predicate + self.iterable = space.iter(w_iterable) + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + while True: + try: + w_obj = self.space.next(self.iterable) + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + w_bool = self.space.call_function(self.w_predicate, w_obj) + if self.space.is_true(w_bool): + return w_obj + +def W_IFilter___new__(space, w_subtype, w_predicate, w_iterable): + return space.wrap(W_IFilter(space, w_predicate, w_iterable)) + +W_IFilter.typedef = TypeDef( + 'ifilter', + __new__ = interp2app(W_IFilter___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), + __iter__ = interp2app(W_IFilter.iter_w, unwrap_spec=['self']), + next = interp2app(W_IFilter.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that filters elements from iterable returning + only those for which the predicate is True. If predicate is + None, return the items that are true. + + Equivalent to : + + def ifilter: + if predicate is None: + predicate = bool + for x in iterable: + if predicate(x): + yield x + """) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 16:47:27 2008 @@ -12,6 +12,7 @@ itertools.repeat(None), itertools.takewhile(bool, []), itertools.dropwhile(bool, []), + itertools.ifilter(None, []), ] for it in iterables: @@ -128,3 +129,30 @@ raises(TypeError, it.next) raises(TypeError, itertools.dropwhile, bool, None) + + def test_ifilter(self): + import itertools + + it = itertools.ifilter(None, []) + raises(StopIteration, it.next) + + it = itertools.ifilter(None, [1, 0, 2, 3, 0]) + for x in [1, 2, 3]: + assert it.next() == x + raises(StopIteration, it.next) + + def is_odd(arg): + return (arg % 2 == 1) + + it = itertools.ifilter(is_odd, [1, 2, 3, 4, 5, 6]) + for x in [1, 3, 5]: + assert it.next() == x + raises(StopIteration, it.next) + + def test_ifilter_wrongargs(self): + import itertools + + it = itertools.ifilter(0, [1]) + raises(TypeError, it.next) + + raises(TypeError, itertools.ifilter, bool, None) From henrikv at codespeak.net Thu Jul 10 16:58:21 2008 From: henrikv at codespeak.net (henrikv at codespeak.net) Date: Thu, 10 Jul 2008 16:58:21 +0200 (CEST) Subject: [pypy-svn] r56420 - in pypy/dist/pypy: module/_rawffi module/_rawffi/test rlib rlib/test rpython/lltypesystem translator/c/test Message-ID: <20080710145821.33D9816A055@codespeak.net> Author: henrikv Date: Thu Jul 10 16:58:20 2008 New Revision: 56420 Modified: pypy/dist/pypy/module/_rawffi/interp_rawffi.py pypy/dist/pypy/module/_rawffi/test/test__rawffi.py pypy/dist/pypy/rlib/libffi.py pypy/dist/pypy/rlib/test/test_libffi.py pypy/dist/pypy/rpython/lltypesystem/ll2ctypes.py pypy/dist/pypy/translator/c/test/test_newgc.py Log: (henrikv) Turn get_libc_name() into a global variable libc_name, and get the name during import/load Modified: pypy/dist/pypy/module/_rawffi/interp_rawffi.py ============================================================================== --- pypy/dist/pypy/module/_rawffi/interp_rawffi.py (original) +++ pypy/dist/pypy/module/_rawffi/interp_rawffi.py Thu Jul 10 16:58:20 2008 @@ -471,6 +471,6 @@ def get_libc(space): try: - return space.wrap(W_CDLL(space, get_libc_name())) + return space.wrap(W_CDLL(space, libc_name)) except OSError, e: raise wrap_oserror(space, e) Modified: pypy/dist/pypy/module/_rawffi/test/test__rawffi.py ============================================================================== --- pypy/dist/pypy/module/_rawffi/test/test__rawffi.py (original) +++ pypy/dist/pypy/module/_rawffi/test/test__rawffi.py Thu Jul 10 16:58:20 2008 @@ -158,11 +158,11 @@ prepare_c_example = staticmethod(prepare_c_example) def setup_class(cls): - from pypy.rlib.libffi import get_libc_name + from pypy.rlib.libffi import libc_name space = gettestobjspace(usemodules=('_rawffi','struct')) cls.space = space cls.w_lib_name = space.wrap(cls.prepare_c_example()) - cls.w_libc_name = space.wrap(get_libc_name()) + cls.w_libc_name = space.wrap(libc_name) if sys.platform == 'win32': cls.w_iswin32 = space.wrap(True) cls.w_libm_name = space.wrap('msvcrt') Modified: pypy/dist/pypy/rlib/libffi.py ============================================================================== --- pypy/dist/pypy/rlib/libffi.py (original) +++ pypy/dist/pypy/rlib/libffi.py Thu Jul 10 16:58:20 2008 @@ -228,8 +228,7 @@ # XXX rffi.cast here... return res - def get_libc_name(): - return ctypes.util.find_library('c') + libc_name = ctypes.util.find_library('c') if _MS_WINDOWS: def dlopen(name): @@ -258,8 +257,7 @@ get_libc_handle = external('get_libc_handle', [], rwin32.HANDLE) - def get_libc_name(): - return rwin32.GetModuleFileName(get_libc_handle()) + libc_name = rwin32.GetModuleFileName(get_libc_handle()) FFI_OK = cConfig.FFI_OK Modified: pypy/dist/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/dist/pypy/rlib/test/test_libffi.py (original) +++ pypy/dist/pypy/rlib/test/test_libffi.py Thu Jul 10 16:58:20 2008 @@ -25,10 +25,10 @@ def test_dlopen(self): py.test.raises(OSError, "dlopen(rffi.str2charp('xxxxxxxxxxxx'))") - assert dlopen(rffi.str2charp(get_libc_name())) + assert dlopen(rffi.str2charp(libc_name)) def get_libc(self): - return CDLL(get_libc_name()) + return CDLL(libc_name) def get_libm(self): if sys.platform == 'win32': @@ -357,11 +357,11 @@ class TestWin32Handles: def setup_class(cls): if sys.platform != 'win32': - py.test.skip("Win-only test") + py.test.skip("Handle to libc library, Win-only test") def test_get_libc_handle(self): handle = get_libc_handle() - print get_libc_name() + print libc_name print hex(handle) assert handle != 0 assert handle % 0x1000 == 0 Modified: pypy/dist/pypy/rpython/lltypesystem/ll2ctypes.py ============================================================================== --- pypy/dist/pypy/rpython/lltypesystem/ll2ctypes.py (original) +++ pypy/dist/pypy/rpython/lltypesystem/ll2ctypes.py Thu Jul 10 16:58:20 2008 @@ -589,6 +589,7 @@ else: return ctypes.util.find_library('c') + libc_name = get_libc_name() # Make sure the name is determined during import, not at runtime standard_c_lib = ctypes.cdll.LoadLibrary(get_libc_name()) # ____________________________________________ Modified: pypy/dist/pypy/translator/c/test/test_newgc.py ============================================================================== --- pypy/dist/pypy/translator/c/test/test_newgc.py (original) +++ pypy/dist/pypy/translator/c/test/test_newgc.py Thu Jul 10 16:58:20 2008 @@ -847,7 +847,7 @@ import gc slong = cast_type_to_ffitype(rffi.LONG) - libc_name = ll2ctypes.get_libc_name() + from pypy.rpython.lltypesystem.ll2ctypes import libc_name def callback(ll_args, ll_res, stuff): gc.collect() From jlg at codespeak.net Thu Jul 10 17:02:55 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Thu, 10 Jul 2008 17:02:55 +0200 (CEST) Subject: [pypy-svn] r56421 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710150255.8F0D716A0F1@codespeak.net> Author: jlg Date: Thu Jul 10 17:02:54 2008 New Revision: 56421 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - ifilterfalse implemented Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Thu Jul 10 17:02:54 2008 @@ -8,6 +8,7 @@ 'count' : 'interp_itertools.W_Count', 'dropwhile' : 'interp_itertools.W_DropWhile', 'ifilter' : 'interp_itertools.W_IFilter', + 'ifilterfalse' : 'interp_itertools.W_IFilterFalse', 'repeat' : 'interp_itertools.W_Repeat', 'takewhile' : 'interp_itertools.W_TakeWhile', } Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 17:02:54 2008 @@ -213,8 +213,7 @@ yield x """) - -class W_IFilter(Wrappable): +class _IFilterBase(Wrappable): def __init__(self, space, w_predicate, w_iterable): self.space = space @@ -234,10 +233,15 @@ except StopIteration: raise OperationError(self.space.w_StopIteration, self.space.w_None) - w_bool = self.space.call_function(self.w_predicate, w_obj) - if self.space.is_true(w_bool): + if self._call_predicate(w_obj): return w_obj + +class W_IFilter(_IFilterBase): + + def _call_predicate(self, w_obj): + return self.space.is_true(self.space.call_function(self.w_predicate, w_obj)) + def W_IFilter___new__(space, w_subtype, w_predicate, w_iterable): return space.wrap(W_IFilter(space, w_predicate, w_iterable)) @@ -259,3 +263,31 @@ if predicate(x): yield x """) + +class W_IFilterFalse(_IFilterBase): + + def _call_predicate(self, w_obj): + return not self.space.is_true(self.space.call_function(self.w_predicate, w_obj)) + +def W_IFilterFalse___new__(space, w_subtype, w_predicate, w_iterable): + return space.wrap(W_IFilterFalse(space, w_predicate, w_iterable)) + +W_IFilterFalse.typedef = TypeDef( + 'ifilterfalse', + __new__ = interp2app(W_IFilterFalse___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), + __iter__ = interp2app(W_IFilterFalse.iter_w, unwrap_spec=['self']), + next = interp2app(W_IFilterFalse.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that filters elements from iterable returning + only those for which the predicate is False. If predicate is + None, return the items that are false. + + Equivalent to : + + def ifilterfalse(predicate, iterable): + if predicate is None: + predicate = bool + for x in iterable: + if not predicate(x): + yield x + """) + Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 17:02:54 2008 @@ -13,6 +13,7 @@ itertools.takewhile(bool, []), itertools.dropwhile(bool, []), itertools.ifilter(None, []), + itertools.ifilterfalse(None, []), ] for it in iterables: @@ -156,3 +157,31 @@ raises(TypeError, it.next) raises(TypeError, itertools.ifilter, bool, None) + + def test_ifilterfalse(self): + import itertools + + it = itertools.ifilterfalse(None, []) + raises(StopIteration, it.next) + + it = itertools.ifilterfalse(None, [1, 0, 2, 3, 0]) + for x in [0, 0]: + assert it.next() == x + raises(StopIteration, it.next) + + def is_odd(arg): + return (arg % 2 == 1) + + it = itertools.ifilterfalse(is_odd, [1, 2, 3, 4, 5, 6]) + for x in [2, 4, 6]: + assert it.next() == x + raises(StopIteration, it.next) + + def test_ifilterfalse_wrongargs(self): + import itertools + + it = itertools.ifilterfalse(0, [1]) + raises(TypeError, it.next) + + raises(TypeError, itertools.ifilterfalse, bool, None) + From bgola at codespeak.net Thu Jul 10 17:07:28 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Thu, 10 Jul 2008 17:07:28 +0200 (CEST) Subject: [pypy-svn] r56422 - pypy/branch/2.5-features/pypy/interpreter/test Message-ID: <20080710150728.65FAB16A14C@codespeak.net> Author: bgola Date: Thu Jul 10 17:07:28 2008 New Revision: 56422 Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py Log: new test for generators Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py Thu Jul 10 17:07:28 2008 @@ -34,6 +34,12 @@ g.next() assert g.send(42) == 42 + def test_generator_raises_typeerror(self): + def f(): + yield 1 + g = f() + raises(TypeError, g.send, 1) + def test_generator_explicit_stopiteration(self): def f(): yield 1 From arigo at codespeak.net Thu Jul 10 18:03:01 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 10 Jul 2008 18:03:01 +0200 (CEST) Subject: [pypy-svn] r56423 - in pypy/dist/pypy/lang/gameboy: . rom/rom1 rom/rom2 rom/rom3 rom/rom4 rom/rom5 rom/rom6 rom/rom7 rom/rom8 rom/rom9 test tool Message-ID: <20080710160301.BC1102A018E@codespeak.net> Author: arigo Date: Thu Jul 10 18:03:00 2008 New Revision: 56423 Modified: pypy/dist/pypy/lang/gameboy/ (props changed) pypy/dist/pypy/lang/gameboy/cartridge.py (props changed) pypy/dist/pypy/lang/gameboy/constants.py (props changed) pypy/dist/pypy/lang/gameboy/cpu.py (props changed) pypy/dist/pypy/lang/gameboy/gameboy.py (props changed) pypy/dist/pypy/lang/gameboy/gameboyTest.py (props changed) pypy/dist/pypy/lang/gameboy/gameboy_implementation.py (props changed) pypy/dist/pypy/lang/gameboy/interrupt.py (props changed) pypy/dist/pypy/lang/gameboy/joypad.py (props changed) pypy/dist/pypy/lang/gameboy/ram.py (props changed) pypy/dist/pypy/lang/gameboy/rom/rom1/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom1/readme.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom2/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom2/readme-1.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom3/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom3/readme-2.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom4/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom4/readme-3.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom5/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom5/readme.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom6/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom6/readme-1.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom7/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom7/readme.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom8/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom8/readme.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/rom/rom9/ (props changed) pypy/dist/pypy/lang/gameboy/rom/rom9/readme.txt (contents, props changed) pypy/dist/pypy/lang/gameboy/serial.py (props changed) pypy/dist/pypy/lang/gameboy/sound.py (props changed) pypy/dist/pypy/lang/gameboy/test/ (props changed) pypy/dist/pypy/lang/gameboy/test/test_cartridge.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_cpu.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_cpu_2.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_gameboy.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_gameboy_implementaton.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_interrupt.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_joypad.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_memory_bank_controller.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_ram.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_register.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_rom.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_serial.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_sound.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_timer.py (props changed) pypy/dist/pypy/lang/gameboy/test/test_video.py (props changed) pypy/dist/pypy/lang/gameboy/timer.py (props changed) pypy/dist/pypy/lang/gameboy/tool/ (props changed) pypy/dist/pypy/lang/gameboy/tool/__init__.py (props changed) pypy/dist/pypy/lang/gameboy/tool/autopath.py (props changed) pypy/dist/pypy/lang/gameboy/video.py (props changed) Log: fixeol Modified: pypy/dist/pypy/lang/gameboy/rom/rom1/readme.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom1/readme.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom1/readme.txt Thu Jul 10 18:03:00 2008 @@ -1,17 +1,17 @@ -============ -Gameboy ROM1 -============ - - -Description ------------ -This is a raw ROM, meaning it contains no valid Gameboy header, just -instructions. Addressing is relative, so the ROM can be loaded anywhere in -memory. - - -Disassembly ------------ -3E 01 LD A, 1 -06 01 LD B, 1 +============ +Gameboy ROM1 +============ + + +Description +----------- +This is a raw ROM, meaning it contains no valid Gameboy header, just +instructions. Addressing is relative, so the ROM can be loaded anywhere in +memory. + + +Disassembly +----------- +3E 01 LD A, 1 +06 01 LD B, 1 80 ADD A, B \ No newline at end of file Modified: pypy/dist/pypy/lang/gameboy/rom/rom2/readme-1.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom2/readme-1.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom2/readme-1.txt Thu Jul 10 18:03:00 2008 @@ -1,17 +1,17 @@ -============ -Gameboy ROM2 -============ - - -Description ------------ -This is a raw ROM, meaning it contains no valid Gameboy header, just -instructions. Addressing is relative, so the ROM can be loaded anywhere in -memory. - - -Disassembly ------------ -3E 00 LD A, 0 -06 00 LD B, 0 +============ +Gameboy ROM2 +============ + + +Description +----------- +This is a raw ROM, meaning it contains no valid Gameboy header, just +instructions. Addressing is relative, so the ROM can be loaded anywhere in +memory. + + +Disassembly +----------- +3E 00 LD A, 0 +06 00 LD B, 0 80 ADD A, B \ No newline at end of file Modified: pypy/dist/pypy/lang/gameboy/rom/rom3/readme-2.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom3/readme-2.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom3/readme-2.txt Thu Jul 10 18:03:00 2008 @@ -1,17 +1,17 @@ -============ -Gameboy ROM3 -============ - - -Specifications --------------- -Filename: noptest.gb -Description: Runs a series of NOP instructions and then at the end performs an - endless loop. -ROM type: NO MBC (32kB ROM, no RAM) - - -Instructions used ------------------ -jp -nop +============ +Gameboy ROM3 +============ + + +Specifications +-------------- +Filename: noptest.gb +Description: Runs a series of NOP instructions and then at the end performs an + endless loop. +ROM type: NO MBC (32kB ROM, no RAM) + + +Instructions used +----------------- +jp +nop Modified: pypy/dist/pypy/lang/gameboy/rom/rom4/readme-3.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom4/readme-3.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom4/readme-3.txt Thu Jul 10 18:03:00 2008 @@ -1,45 +1,45 @@ -============ -Gameboy ROM4 -============ - - -Specifications --------------- -Filename: rom4.gb -Description: This ROM is created to test the DI and HALT instructions. It also - uses the LCDC register to turn off the screen. -ROM type: NO MBC (32kB ROM, no RAM) - - -Instructions used ------------------ -ld -jp -nop -res -di -halt - - -Memory registers used ---------------------- -$ff40 (LCDC register, used to turn of screen) - - -Short information ------------------ -HALT stops the CPU until an interrupt occurs. Nintendo recommends using this -command in your main game loop in order to save battery power while the CPU has -nothing else to do. - -When an interrupt occurs while in HALT, the CPU starts back up and pushes the -Program Counter onto the stack before servicing the interrupt(s). Except it -doesn't push the address after HALT as one might expect but rather the address -of HALT itself. - -Nintendo also recommends that you put a NOP after HALT commands. The reason for -this is that the Program Counter will not increment properly (CPU bug) if you -try to do a HALT while IME = 0 and an interrupt is pending. A single-byte -instruction immediately following HALT will get executed twice if IME = 0 and an -interrupt is pending. If the instruction following HALT is a multi-byte -instruction then the game could hang or registers could get scrambled. +============ +Gameboy ROM4 +============ + + +Specifications +-------------- +Filename: rom4.gb +Description: This ROM is created to test the DI and HALT instructions. It also + uses the LCDC register to turn off the screen. +ROM type: NO MBC (32kB ROM, no RAM) + + +Instructions used +----------------- +ld +jp +nop +res +di +halt + + +Memory registers used +--------------------- +$ff40 (LCDC register, used to turn of screen) + + +Short information +----------------- +HALT stops the CPU until an interrupt occurs. Nintendo recommends using this +command in your main game loop in order to save battery power while the CPU has +nothing else to do. + +When an interrupt occurs while in HALT, the CPU starts back up and pushes the +Program Counter onto the stack before servicing the interrupt(s). Except it +doesn't push the address after HALT as one might expect but rather the address +of HALT itself. + +Nintendo also recommends that you put a NOP after HALT commands. The reason for +this is that the Program Counter will not increment properly (CPU bug) if you +try to do a HALT while IME = 0 and an interrupt is pending. A single-byte +instruction immediately following HALT will get executed twice if IME = 0 and an +interrupt is pending. If the instruction following HALT is a multi-byte +instruction then the game could hang or registers could get scrambled. Modified: pypy/dist/pypy/lang/gameboy/rom/rom5/readme.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom5/readme.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom5/readme.txt Thu Jul 10 18:03:00 2008 @@ -1,35 +1,35 @@ -============ -Gameboy ROM5 -============ - - -Specifications --------------- -Filename: rom5.gb -Description: This ROM is created to test the conditional jump instructions. It - also includes an invalid opcode ($DD) which is used for debugging - purposes! -ROM type: NO MBC (32kB ROM, no RAM) - - -Special notes -------------- -IMPORTANT: This ROM contains an invalid opcode ($DD) which we use to trigger the -emulator into a "breakpoint". When executing this ROM, simply treat the $DD -bytes as NOP ($00) instructions, but also print out the registers and CPU -status. - -This ROM also uses the carry and zero flags, so make sure that the work correctly. - - -Instructions used ------------------ -ld -jp -jr -inc -dec -nop -di -halt +============ +Gameboy ROM5 +============ + + +Specifications +-------------- +Filename: rom5.gb +Description: This ROM is created to test the conditional jump instructions. It + also includes an invalid opcode ($DD) which is used for debugging + purposes! +ROM type: NO MBC (32kB ROM, no RAM) + + +Special notes +------------- +IMPORTANT: This ROM contains an invalid opcode ($DD) which we use to trigger the +emulator into a "breakpoint". When executing this ROM, simply treat the $DD +bytes as NOP ($00) instructions, but also print out the registers and CPU +status. + +This ROM also uses the carry and zero flags, so make sure that the work correctly. + + +Instructions used +----------------- +ld +jp +jr +inc +dec +nop +di +halt $DD \ No newline at end of file Modified: pypy/dist/pypy/lang/gameboy/rom/rom6/readme-1.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom6/readme-1.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom6/readme-1.txt Thu Jul 10 18:03:00 2008 @@ -1,31 +1,31 @@ -============ -Gameboy ROM6 -============ - - -Specifications --------------- -Filename: rom6.gb -Description: This ROM is created to test the implementation of ROM cartridge - with an MBC1 controller. -ROM type: MBC1 + RAM (64kB ROM, 32kB RAM) - - -Special notes -------------- -This ROM uses call and ret instructions, so this means that the emulator will -also need to implement and support a stack. - -Instructions used ------------------ -ld -jp -jr -inc -dec -nop -di -halt -$DD -call +============ +Gameboy ROM6 +============ + + +Specifications +-------------- +Filename: rom6.gb +Description: This ROM is created to test the implementation of ROM cartridge + with an MBC1 controller. +ROM type: MBC1 + RAM (64kB ROM, 32kB RAM) + + +Special notes +------------- +This ROM uses call and ret instructions, so this means that the emulator will +also need to implement and support a stack. + +Instructions used +----------------- +ld +jp +jr +inc +dec +nop +di +halt +$DD +call ret \ No newline at end of file Modified: pypy/dist/pypy/lang/gameboy/rom/rom7/readme.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom7/readme.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom7/readme.txt Thu Jul 10 18:03:00 2008 @@ -1,10 +1,10 @@ -============ -Gameboy ROM7 -============ - - -Specifications --------------- -Filename: rom7.gb -Description: This ROM is created to test the screen background. +============ +Gameboy ROM7 +============ + + +Specifications +-------------- +Filename: rom7.gb +Description: This ROM is created to test the screen background. ROM type: NOMBC \ No newline at end of file Modified: pypy/dist/pypy/lang/gameboy/rom/rom8/readme.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom8/readme.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom8/readme.txt Thu Jul 10 18:03:00 2008 @@ -1,10 +1,10 @@ -============ -Gameboy ROM8 -============ - - -Specifications --------------- -Filename: rom8.gb -Description: This ROM is created to test graphics background of a Gameboy. +============ +Gameboy ROM8 +============ + + +Specifications +-------------- +Filename: rom8.gb +Description: This ROM is created to test graphics background of a Gameboy. ROM type: MBC1 + RAM (64kB ROM, 32kB RAM) \ No newline at end of file Modified: pypy/dist/pypy/lang/gameboy/rom/rom9/readme.txt ============================================================================== --- pypy/dist/pypy/lang/gameboy/rom/rom9/readme.txt (original) +++ pypy/dist/pypy/lang/gameboy/rom/rom9/readme.txt Thu Jul 10 18:03:00 2008 @@ -1,10 +1,10 @@ -============ -Gameboy ROM9 -============ - - -Specifications --------------- -Filename: rom9.gb -Description: This ROM is created to test graphics background of a Gameboy and sprites. +============ +Gameboy ROM9 +============ + + +Specifications +-------------- +Filename: rom9.gb +Description: This ROM is created to test graphics background of a Gameboy and sprites. ROM type: MBC1 + RAM (64kB ROM, 32kB RAM) \ No newline at end of file From arigo at codespeak.net Thu Jul 10 18:28:17 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 10 Jul 2008 18:28:17 +0200 (CEST) Subject: [pypy-svn] r56424 - pypy/dist/pypy/module/itertools Message-ID: <20080710162817.66E46169FB6@codespeak.net> Author: arigo Date: Thu Jul 10 18:28:15 2008 New Revision: 56424 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py Log: Remove unnecessary catching. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 18:28:15 2008 @@ -122,11 +122,7 @@ if self.stopped: raise OperationError(self.space.w_StopIteration, self.space.w_None) - try: - w_obj = self.space.next(self.iterable) - except StopIteration: - raise OperationError(self.space.w_StopIteration, self.space.w_None) - + w_obj = self.space.next(self.iterable) # may raise a w_StopIteration w_bool = self.space.call_function(self.w_predicate, w_obj) if not self.space.is_true(w_bool): self.stopped = True @@ -169,17 +165,10 @@ def next_w(self): if self.started: - try: - w_obj = self.space.next(self.iterable) - except StopIteration: - raise OperationError(self.space.w_StopIteration, self.space.w_None) + w_obj = self.space.next(self.iterable) # may raise w_StopIteration else: while True: - try: - w_obj = self.space.next(self.iterable) - except StopIteration: - raise OperationError(self.space.w_StopIteration, self.space.w_None) - + w_obj = self.space.next(self.iterable) # may raise w_StopIter w_bool = self.space.call_function(self.w_predicate, w_obj) if not self.space.is_true(w_bool): self.started = True @@ -228,11 +217,7 @@ def next_w(self): while True: - try: - w_obj = self.space.next(self.iterable) - except StopIteration: - raise OperationError(self.space.w_StopIteration, self.space.w_None) - + w_obj = self.space.next(self.iterable) # may raise w_StopIteration if self._call_predicate(w_obj): return w_obj From jlg at codespeak.net Thu Jul 10 19:14:45 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Thu, 10 Jul 2008 19:14:45 +0200 (CEST) Subject: [pypy-svn] r56425 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710171445.6F2BC169ED3@codespeak.net> Author: jlg Date: Thu Jul 10 19:14:43 2008 New Revision: 56425 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - islice partialy implemented Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Thu Jul 10 19:14:43 2008 @@ -9,6 +9,7 @@ 'dropwhile' : 'interp_itertools.W_DropWhile', 'ifilter' : 'interp_itertools.W_IFilter', 'ifilterfalse' : 'interp_itertools.W_IFilterFalse', + 'islice' : 'interp_itertools.W_ISlice', 'repeat' : 'interp_itertools.W_Repeat', 'takewhile' : 'interp_itertools.W_TakeWhile', } Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 19:14:43 2008 @@ -276,3 +276,60 @@ yield x """) +class W_ISlice (Wrappable): + + def __init__(self, space, w_iterable, start, stop, step): + self.iterable = space.iter(w_iterable) + self.space = space + if stop == -1: + stop = start + start = 0 + + if step == -1: + step = 1 + + self.start = start + self.stop = stop + self.step = step + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if self.stop <= 0: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + if self.start >= 0: + skip = self.start + self.start = -1 + else: + skip = self.step - 1 + + try: + while skip > 0: + self.space.next(self.iterable) + skip -= 1 + self.stop -= 1 + + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + try: + w_obj = self.space.next(self.iterable) + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + self.stop -= 1 + return w_obj + +def W_ISlice___new__(space, w_subtype, w_iterable, start, stop, step): + # TODO varible arguments number not implemented (optional start, step) + return space.wrap(W_ISlice(space, w_iterable, start, stop, step)) + +W_ISlice.typedef = TypeDef( + 'islice', + __new__ = interp2app(W_ISlice___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, int, int, int]), + __iter__ = interp2app(W_ISlice.iter_w, unwrap_spec=['self']), + next = interp2app(W_ISlice.next_w, unwrap_spec=['self']), + __doc__ = "") + Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 19:14:43 2008 @@ -14,6 +14,7 @@ itertools.dropwhile(bool, []), itertools.ifilter(None, []), itertools.ifilterfalse(None, []), + itertools.islice([], 0, -1, -1), ] for it in iterables: @@ -185,3 +186,44 @@ raises(TypeError, itertools.ifilterfalse, bool, None) + def test_islice(self): + import itertools + + it = itertools.islice([], 0, -1, -1) + raises(StopIteration, it.next) + + it = itertools.islice([1, 2, 3], 0, -1, -1) + raises(StopIteration, it.next) + + it = itertools.islice([1, 2, 3, 4, 5], 3, -1, -1) + for x in [1, 2, 3]: + assert it.next() == x + raises(StopIteration, it.next) + + it = itertools.islice([1, 2, 3, 4, 5], 3, -1, -1) + for x in [1, 2, 3]: + assert it.next() == x + raises(StopIteration, it.next) + + it = itertools.islice([1, 2, 3, 4, 5], 1, 3, -1) + for x in [2, 3]: + assert it.next() == x + raises(StopIteration, it.next) + + it = itertools.islice([1, 2, 3, 4, 5], 0, 3, 2) + for x in [1, 3]: + assert it.next() == x + raises(StopIteration, it.next) + + def test_islice_overflow(self): + import itertools + import sys + + raises(OverflowError, itertools.islice, [], sys.maxint + 1, -1, -1) + + def test_islice_wrongargs(self): + import itertools + + raises(TypeError, itertools.islice, [], None, -1, -1) + raises(TypeError, itertools.islice, None, 0, -1, -1) + From cfbolz at codespeak.net Thu Jul 10 23:48:03 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 10 Jul 2008 23:48:03 +0200 (CEST) Subject: [pypy-svn] r56426 - pypy/dist/pypy/module/itertools Message-ID: <20080710214803.E6CE7168546@codespeak.net> Author: cfbolz Date: Thu Jul 10 23:48:02 2008 New Revision: 56426 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py Log: Use a flag instead of a method. No code duplication and no method call. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 23:48:02 2008 @@ -218,14 +218,14 @@ def next_w(self): while True: w_obj = self.space.next(self.iterable) # may raise w_StopIteration - if self._call_predicate(w_obj): + w_pred = self.space.call_function(self.w_predicate, w_obj) + pred = self.space.is_true(w_pred) + if pred ^ self.reverse: return w_obj class W_IFilter(_IFilterBase): - - def _call_predicate(self, w_obj): - return self.space.is_true(self.space.call_function(self.w_predicate, w_obj)) + reverse = False def W_IFilter___new__(space, w_subtype, w_predicate, w_iterable): return space.wrap(W_IFilter(space, w_predicate, w_iterable)) @@ -250,9 +250,7 @@ """) class W_IFilterFalse(_IFilterBase): - - def _call_predicate(self, w_obj): - return not self.space.is_true(self.space.call_function(self.w_predicate, w_obj)) + reverse = True def W_IFilterFalse___new__(space, w_subtype, w_predicate, w_iterable): return space.wrap(W_IFilterFalse(space, w_predicate, w_iterable)) From cfbolz at codespeak.net Thu Jul 10 23:59:46 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 10 Jul 2008 23:59:46 +0200 (CEST) Subject: [pypy-svn] r56427 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080710215946.B8DBB49801A@codespeak.net> Author: cfbolz Date: Thu Jul 10 23:59:46 2008 New Revision: 56427 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: Fix broken __new__ implementations. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Thu Jul 10 23:59:46 2008 @@ -32,7 +32,9 @@ """ Create a new count object and call its initializer. """ - return space.wrap(W_Count(space, firstval)) + result = space.allocate_instance(W_Count, w_subtype) + W_Count.__init__(result, space, firstval) + return space.wrap(result) W_Count.typedef = TypeDef( 'count', @@ -83,7 +85,9 @@ """ Create a new repeat object and call its initializer. """ - return space.wrap(W_Repeat(space, w_obj, w_times)) + result = space.allocate_instance(W_Repeat, w_subtype) + W_Repeat.__init__(result, space, w_obj, w_times) + return space.wrap(result) W_Repeat.typedef = TypeDef( 'repeat', @@ -131,7 +135,9 @@ return w_obj def W_TakeWhile___new__(space, w_subtype, w_predicate, w_iterable): - return space.wrap(W_TakeWhile(space, w_predicate, w_iterable)) + result = space.allocate_instance(W_TakeWhile, w_subtype) + W_TakeWhile.__init__(result, space, w_predicate, w_iterable) + return space.wrap(result) W_TakeWhile.typedef = TypeDef( @@ -177,7 +183,9 @@ return w_obj def W_DropWhile___new__(space, w_subtype, w_predicate, w_iterable): - return space.wrap(W_DropWhile(space, w_predicate, w_iterable)) + result = space.allocate_instance(W_DropWhile, w_subtype) + W_DropWhile.__init__(result, space, w_predicate, w_iterable) + return space.wrap(result) W_DropWhile.typedef = TypeDef( @@ -228,7 +236,9 @@ reverse = False def W_IFilter___new__(space, w_subtype, w_predicate, w_iterable): - return space.wrap(W_IFilter(space, w_predicate, w_iterable)) + result = space.allocate_instance(W_IFilter, w_subtype) + W_IFilter.__init__(result, space, w_predicate, w_iterable) + return space.wrap(result) W_IFilter.typedef = TypeDef( 'ifilter', @@ -253,7 +263,9 @@ reverse = True def W_IFilterFalse___new__(space, w_subtype, w_predicate, w_iterable): - return space.wrap(W_IFilterFalse(space, w_predicate, w_iterable)) + result = space.allocate_instance(W_IFilterFalse, w_subtype) + W_IFilterFalse.__init__(result, space, w_predicate, w_iterable) + return space.wrap(result) W_IFilterFalse.typedef = TypeDef( 'ifilterfalse', @@ -322,7 +334,9 @@ def W_ISlice___new__(space, w_subtype, w_iterable, start, stop, step): # TODO varible arguments number not implemented (optional start, step) - return space.wrap(W_ISlice(space, w_iterable, start, stop, step)) + result = space.allocate_instance(W_ISlice, w_subtype) + W_ISlice.__init__(result, space, w_iterable, start, stop, step) + return space.wrap(result) W_ISlice.typedef = TypeDef( 'islice', Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Thu Jul 10 23:59:46 2008 @@ -227,3 +227,20 @@ raises(TypeError, itertools.islice, [], None, -1, -1) raises(TypeError, itertools.islice, None, 0, -1, -1) + def test_subclassing(self): + import itertools + # not sure how useful this actually is, but CPython implements it + iterables = [ + (itertools.count, ()), + (itertools.repeat, (None,)), + (itertools.takewhile, (bool, [])), + (itertools.dropwhile, (bool, [])), + (itertools.ifilter, (None, [])), + (itertools.ifilterfalse, (None, [])), + (itertools.islice, ([], 0, -1, -1)), + ] + for cls, args in iterables: + class A(cls): + pass + a = A(*args) + assert isinstance(a, A) From fijal at codespeak.net Fri Jul 11 08:22:49 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Fri, 11 Jul 2008 08:22:49 +0200 (CEST) Subject: [pypy-svn] r56429 - pypy/dist/pypy/lib Message-ID: <20080711062249.8E8A716857A@codespeak.net> Author: fijal Date: Fri Jul 11 08:22:46 2008 New Revision: 56429 Modified: pypy/dist/pypy/lib/ (props changed) Log: Add pysqlite2 to svn:externals here. I would like hg:externals, but well... From adurdin at codespeak.net Fri Jul 11 08:51:40 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 08:51:40 +0200 (CEST) Subject: [pypy-svn] r56430 - pypy/dist/pypy/module/itertools Message-ID: <20080711065140.99F692A00E3@codespeak.net> Author: adurdin Date: Fri Jul 11 08:51:39 2008 New Revision: 56430 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py Log: (adurdin, jlg) removing more naively redundant exception handling from itertools. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 08:51:39 2008 @@ -315,20 +315,12 @@ else: skip = self.step - 1 - try: - while skip > 0: - self.space.next(self.iterable) - skip -= 1 - self.stop -= 1 - - except StopIteration: - raise OperationError(self.space.w_StopIteration, self.space.w_None) - - try: - w_obj = self.space.next(self.iterable) - except StopIteration: - raise OperationError(self.space.w_StopIteration, self.space.w_None) + while skip > 0: + self.space.next(self.iterable) + skip -= 1 + self.stop -= 1 + w_obj = self.space.next(self.iterable) self.stop -= 1 return w_obj From fijal at codespeak.net Fri Jul 11 09:19:52 2008 From: fijal at codespeak.net (fijal at codespeak.net) Date: Fri, 11 Jul 2008 09:19:52 +0200 (CEST) Subject: [pypy-svn] r56431 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080711071952.A50002A0152@codespeak.net> Author: fijal Date: Fri Jul 11 09:19:52 2008 New Revision: 56431 Modified: pypy/extradoc/sprintinfo/post-ep2008/planning.txt Log: (everybody) planning for today Modified: pypy/extradoc/sprintinfo/post-ep2008/planning.txt ============================================================================== --- pypy/extradoc/sprintinfo/post-ep2008/planning.txt (original) +++ pypy/extradoc/sprintinfo/post-ep2008/planning.txt Fri Jul 11 09:19:52 2008 @@ -9,20 +9,26 @@ Tasks: - try out Python programs and fix them or fix PyPy or fix - performance bottlenecks (geoff, henrik, david, fijal around) + performance bottlenecks. (fijal, henrikv) + * Django seems to be running. + * Fixed _rawffi for Mac OS X + * CherryPy works out of the box - - support for profiling the calls to builtins (stephan, fijal - around) + - support for profiling the calls to builtins (IN-PROGRESS) + (stephan, anto) - fix the default options of "translate.py", make "--help" show only the reasonable options, remove "--faassen", etc. (samuele, others for discussion) - dotviewer improvements (expanding & collapsing, etc.) - and eventually make it a separate project (dinu, marius) + and eventually make it a separate project + * Having fun + * Minor progress + * IN-PROGRESS (dinu, marius) - start a tutorial on how to write a language interpreter - in RPython with PyPy (jonathan, ...) + in RPython with PyPy - port the stackless transform to ootypesystem @@ -30,7 +36,7 @@ - profile & speedup pypy-jvm - - more extension modules, with ctypes or rpython (andrew, - jakub, gasper, ...) + - more extension modules, with ctypes or rpython + * itertools IN-PROGRESS (adurdin, jakub, anto around) - general wizardry (anto, arigo) From henrikv at codespeak.net Fri Jul 11 09:56:47 2008 From: henrikv at codespeak.net (henrikv at codespeak.net) Date: Fri, 11 Jul 2008 09:56:47 +0200 (CEST) Subject: [pypy-svn] r56432 - pypy/dist/pypy/lib/ctypes Message-ID: <20080711075647.9B1D416852E@codespeak.net> Author: henrikv Date: Fri Jul 11 09:56:46 2008 New Revision: 56432 Modified: pypy/dist/pypy/lib/ctypes/__init__.py Log: (henrikv,fijal,Bill Janssen) Use sys.platform instead of gestalt, as it is not distribution dependant. ctypes tests pass on Mac OSX Modified: pypy/dist/pypy/lib/ctypes/__init__.py ============================================================================== --- pypy/dist/pypy/lib/ctypes/__init__.py (original) +++ pypy/dist/pypy/lib/ctypes/__init__.py Fri Jul 11 09:56:46 2008 @@ -24,19 +24,13 @@ DEFAULT_MODE = RTLD_LOCAL if _os.name == "posix" and _sys.platform == "darwin": - import gestalt + import platform - # gestalt.gestalt("sysv") returns the version number of the - # currently active system file as BCD. - # On OS X 10.4.6 -> 0x1046 - # On OS X 10.2.8 -> 0x1028 - # See also http://www.rgaros.nl/gestalt/ - # # On OS X 10.3, we use RTLD_GLOBAL as default mode # because RTLD_LOCAL does not work at least on some # libraries. - if gestalt.gestalt("sysv") < 0x1040: + if [10, 4] > [int(x) for x in platform.release().split('.')[:2]]: DEFAULT_MODE = RTLD_GLOBAL from _ctypes import FUNCFLAG_CDECL as _FUNCFLAG_CDECL, \ From hpk at codespeak.net Fri Jul 11 09:58:18 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Fri, 11 Jul 2008 09:58:18 +0200 (CEST) Subject: [pypy-svn] r56433 - pypy/dist/pypy/lib/ctypes Message-ID: <20080711075818.47D8D168531@codespeak.net> Author: hpk Date: Fri Jul 11 09:58:17 2008 New Revision: 56433 Modified: pypy/dist/pypy/lib/ctypes/__init__.py Log: check the version of darwin, not osx (see http://mail.python.org/pipermail/pythonmac-sig/2007-September/019216.html) Modified: pypy/dist/pypy/lib/ctypes/__init__.py ============================================================================== --- pypy/dist/pypy/lib/ctypes/__init__.py (original) +++ pypy/dist/pypy/lib/ctypes/__init__.py Fri Jul 11 09:58:17 2008 @@ -30,8 +30,8 @@ # because RTLD_LOCAL does not work at least on some # libraries. - if [10, 4] > [int(x) for x in platform.release().split('.')[:2]]: - DEFAULT_MODE = RTLD_GLOBAL + if int(platform.release().split('.')[0]) < 8: + DEFAULT_MODE = RTLD_GLOBAL from _ctypes import FUNCFLAG_CDECL as _FUNCFLAG_CDECL, \ FUNCFLAG_PYTHONAPI as _FUNCFLAG_PYTHONAPI From jlg at codespeak.net Fri Jul 11 09:58:50 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Fri, 11 Jul 2008 09:58:50 +0200 (CEST) Subject: [pypy-svn] r56434 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711075850.0FF2916852E@codespeak.net> Author: jlg Date: Fri Jul 11 09:58:49 2008 New Revision: 56434 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - islice conflict resolved Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 09:58:49 2008 @@ -286,18 +286,39 @@ yield x """) -class W_ISlice (Wrappable): - - def __init__(self, space, w_iterable, start, stop, step): +class W_ISlice(Wrappable): + def __init__(self, space, w_iterable, w_startstop, args_w): self.iterable = space.iter(w_iterable) self.space = space - if stop == -1: - stop = start + + num_args = len(args_w) + + if num_args == 0: start = 0 + w_stop = w_startstop + elif num_args <= 2: + start = space.int_w(w_startstop) + w_stop = args_w[0] + else: + raise OperationError(space.w_TypeError, space.wrap("islice() takes at most 4 arguments (" + str(num_args) + " given)")) + + if space.is_w(w_stop, space.w_None): + stop = None + else: + stop = space.int_w(w_stop) - if step == -1: + if num_args == 2: + step = space.int_w(args_w[1]) + else: step = 1 + if start < 0: + raise OperationError(space.w_ValueError, space.wrap("Indicies for islice() must be non-negative integers.")) + if stop is not None and stop < 0: + raise OperationError(space.w_ValueError, space.wrap("Stop argument must be a non-negative integer or None.")) + if step < 1: + raise OperationError(space.w_ValueError, space.wrap("Step must be one or lager for islice().")) + self.start = start self.stop = stop self.step = step @@ -306,7 +327,7 @@ return self.space.wrap(self) def next_w(self): - if self.stop <= 0: + if self.stop is not None and self.stop <= 0: raise OperationError(self.space.w_StopIteration, self.space.w_None) if self.start >= 0: @@ -318,22 +339,34 @@ while skip > 0: self.space.next(self.iterable) skip -= 1 - self.stop -= 1 + if self.stop is not None: + self.stop -= 1 w_obj = self.space.next(self.iterable) - self.stop -= 1 + if self.stop is not None: + self.stop -= 1 return w_obj -def W_ISlice___new__(space, w_subtype, w_iterable, start, stop, step): - # TODO varible arguments number not implemented (optional start, step) +def W_ISlice___new__(space, w_subtype, w_iterable, w_startstop, args_w): result = space.allocate_instance(W_ISlice, w_subtype) - W_ISlice.__init__(result, space, w_iterable, start, stop, step) + W_ISlice.__init__(result, space, w_iterable, w_startstop, args_w) return space.wrap(result) W_ISlice.typedef = TypeDef( 'islice', - __new__ = interp2app(W_ISlice___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, int, int, int]), + __new__ = interp2app(W_ISlice___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root, 'args_w']), __iter__ = interp2app(W_ISlice.iter_w, unwrap_spec=['self']), next = interp2app(W_ISlice.next_w, unwrap_spec=['self']), - __doc__ = "") + __doc__ = """Make an iterator that returns selected elements from the + iterable. If start is non-zero, then elements from the iterable + are skipped until start is reached. Afterward, elements are + returned consecutively unless step is set higher than one which + results in items being skipped. If stop is None, then iteration + continues until the iterator is exhausted, if at all; otherwise, + it stops at the specified position. Unlike regular slicing, + islice() does not support negative values for start, stop, or + step. Can be used to extract related fields from data where the + internal structure has been flattened (for example, a multi-line + report may list a name field on every third line). + """) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 09:58:49 2008 @@ -14,7 +14,7 @@ itertools.dropwhile(bool, []), itertools.ifilter(None, []), itertools.ifilterfalse(None, []), - itertools.islice([], 0, -1, -1), + itertools.islice([], 0), ] for it in iterables: @@ -189,23 +189,23 @@ def test_islice(self): import itertools - it = itertools.islice([], 0, -1, -1) + it = itertools.islice([], 0) raises(StopIteration, it.next) - it = itertools.islice([1, 2, 3], 0, -1, -1) + it = itertools.islice([1, 2, 3], 0) raises(StopIteration, it.next) - it = itertools.islice([1, 2, 3, 4, 5], 3, -1, -1) + it = itertools.islice([1, 2, 3, 4, 5], 3) for x in [1, 2, 3]: assert it.next() == x raises(StopIteration, it.next) - it = itertools.islice([1, 2, 3, 4, 5], 3, -1, -1) + it = itertools.islice([1, 2, 3, 4, 5], 3) for x in [1, 2, 3]: assert it.next() == x raises(StopIteration, it.next) - it = itertools.islice([1, 2, 3, 4, 5], 1, 3, -1) + it = itertools.islice([1, 2, 3, 4, 5], 1, 3) for x in [2, 3]: assert it.next() == x raises(StopIteration, it.next) @@ -215,17 +215,32 @@ assert it.next() == x raises(StopIteration, it.next) + it = itertools.islice([1, 2, 3], 0, None) + for x in [1, 2, 3]: + assert it.next() == x + raises(StopIteration, it.next) + def test_islice_overflow(self): import itertools import sys - raises(OverflowError, itertools.islice, [], sys.maxint + 1, -1, -1) + raises(OverflowError, itertools.islice, [], sys.maxint + 1) def test_islice_wrongargs(self): import itertools - raises(TypeError, itertools.islice, [], None, -1, -1) - raises(TypeError, itertools.islice, None, 0, -1, -1) + raises(TypeError, itertools.islice, None, 0) + + raises(ValueError, itertools.islice, [], -1) + + raises(ValueError, itertools.islice, [], -1, 0) + raises(ValueError, itertools.islice, [], 0, -1) + + raises(ValueError, itertools.islice, [], 0, 0, -1) + raises(ValueError, itertools.islice, [], 0, 0, 0) + raises(TypeError, itertools.islice, [], 0, 0, None) + + raises(TypeError, itertools.islice, [], 0, 0, 0, 0) def test_subclassing(self): import itertools @@ -237,10 +252,11 @@ (itertools.dropwhile, (bool, [])), (itertools.ifilter, (None, [])), (itertools.ifilterfalse, (None, [])), - (itertools.islice, ([], 0, -1, -1)), + (itertools.islice, ([], 0)), ] for cls, args in iterables: class A(cls): pass a = A(*args) assert isinstance(a, A) + From arigo at codespeak.net Fri Jul 11 10:00:52 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 11 Jul 2008 10:00:52 +0200 (CEST) Subject: [pypy-svn] r56435 - pypy/dist/pypy/doc/config Message-ID: <20080711080052.68390168534@codespeak.net> Author: arigo Date: Fri Jul 11 10:00:51 2008 New Revision: 56435 Added: pypy/dist/pypy/doc/config/objspace.usemodules.itertools.txt (contents, props changed) Log: Document this option. Added: pypy/dist/pypy/doc/config/objspace.usemodules.itertools.txt ============================================================================== --- (empty file) +++ pypy/dist/pypy/doc/config/objspace.usemodules.itertools.txt Fri Jul 11 10:00:51 2008 @@ -0,0 +1,2 @@ +Use the interp-level 'itertools' module. +If not included, a slower app-level version of itertools is used. From adurdin at codespeak.net Fri Jul 11 10:09:16 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 10:09:16 +0200 (CEST) Subject: [pypy-svn] r56436 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711080916.58CBA169E1A@codespeak.net> Author: adurdin Date: Fri Jul 11 10:09:15 2008 New Revision: 56436 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) Added test for presence of docstrings. Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Fri Jul 11 10:09:15 2008 @@ -2,7 +2,27 @@ from pypy.interpreter.mixedmodule import MixedModule class Module(MixedModule): - """An itertools module.""" + """Functional tools for creating and using iterators. + + Infinite iterators: + count([n]) --> n, n+1, n+2, ... + cycle(p) --> p0, p1, ... plast, p0, p1, ... + repeat(elem [,n]) --> elem, elem, elem, ... endlessly or up to n times + + Iterators terminating on the shortest input sequence: + izip(p, q, ...) --> (p[0], q[0]), (p[1], q[1]), ... + ifilter(pred, seq) --> elements of seq where pred(elem) is True + ifilterfalse(pred, seq) --> elements of seq where pred(elem) is False + islice(seq, [start,] stop [, step]) --> elements from + seq[start:stop:step] + imap(fun, p, q, ...) --> fun(p0, q0), fun(p1, q1), ... + starmap(fun, seq) --> fun(*seq[0]), fun(*seq[1]), ... + tee(it, n=2) --> (it1, it2 , ... itn) splits one iterator into n + chain(p, q, ...) --> p0, p1, ... plast, q0, q1, ... + takewhile(pred, seq) --> seq[0], seq[1], until pred fails + dropwhile(pred, seq) --> seq[n], seq[n+1], starting when pred fails + groupby(iterable[, keyfunc]) --> sub-iterators grouped by value of keyfunc(v) + """ interpleveldefs = { 'count' : 'interp_itertools.W_Count', Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 10:09:15 2008 @@ -242,6 +242,23 @@ raises(TypeError, itertools.islice, [], 0, 0, 0, 0) + def test_docstrings(self): + import itertools + + assert itertools.__doc__ != "" + methods = [ + itertools.count, + itertools.repeat, + itertools.takewhile, + itertools.dropwhile, + itertools.ifilter, + itertools.ifilterfalse, + itertools.islice, + ] + for method in methods: + assert method.__doc__ != "" + + def test_subclassing(self): import itertools # not sure how useful this actually is, but CPython implements it From antocuni at codespeak.net Fri Jul 11 11:06:22 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 11 Jul 2008 11:06:22 +0200 (CEST) Subject: [pypy-svn] r56437 - pypy/branch/builtin-profiling Message-ID: <20080711090622.C17F069804B@codespeak.net> Author: antocuni Date: Fri Jul 11 11:06:21 2008 New Revision: 56437 Added: pypy/branch/builtin-profiling/ - copied from r56436, pypy/dist/ Log: (stephan, anto) a branch to try to add profiling info when calling builtin functions. We do it in a branch to make sure we don't slow things too much From adurdin at codespeak.net Fri Jul 11 11:06:50 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 11:06:50 +0200 (CEST) Subject: [pypy-svn] r56438 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711090650.F0011698084@codespeak.net> Author: adurdin Date: Fri Jul 11 11:06:50 2008 New Revision: 56438 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) Implemented interp_itertools.chain Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Fri Jul 11 11:06:50 2008 @@ -25,6 +25,7 @@ """ interpleveldefs = { + 'chain' : 'interp_itertools.W_Chain', 'count' : 'interp_itertools.W_Count', 'dropwhile' : 'interp_itertools.W_DropWhile', 'ifilter' : 'interp_itertools.W_IFilter', Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 11:06:50 2008 @@ -370,3 +370,70 @@ report may list a name field on every third line). """) + +class W_Chain(Wrappable): + def __init__(self, space, args_w): + self.space = space + iterators_w = [] + for i, iterable_w in enumerate(args_w): + try: + iterator_w = space.iter(iterable_w) + except OperationError, e: + if e.match(self.space, self.space.w_TypeError): + raise OperationError(space.w_TypeError, space.wrap("chain argument #" + str(i + 1) + " must support iteration")) + else: + raise + else: + iterators_w.append(iterator_w) + self.iterators_w = iter(iterators_w) + self.started = False + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if not self.started: + try: + self.w_it = self.iterators_w.next() + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + else: + self.started = True + + while True: + try: + w_obj = self.space.next(self.w_it) + except OperationError, e: + if e.match(self.space, self.space.w_StopIteration): + try: + self.w_it = self.iterators_w.next() + except StopIteration: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + else: + raise + else: + break + return w_obj + +def W_Chain___new__(space, w_subtype, args_w): + result = space.allocate_instance(W_Chain, w_subtype) + W_Chain.__init__(result, space, args_w) + return space.wrap(result) + +W_Chain.typedef = TypeDef( + 'chain', + __new__ = interp2app(W_Chain___new__, unwrap_spec=[ObjSpace, W_Root, 'args_w']), + __iter__ = interp2app(W_Chain.iter_w, unwrap_spec=['self']), + next = interp2app(W_Chain.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that returns elements from the first iterable + until it is exhausted, then proceeds to the next iterable, until + all of the iterables are exhausted. Used for treating consecutive + sequences as a single sequence. + + Equivalent to : + + def chain(*iterables): + for it in iterables: + for element in it: + yield element + """) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 11:06:50 2008 @@ -15,6 +15,7 @@ itertools.ifilter(None, []), itertools.ifilterfalse(None, []), itertools.islice([], 0), + itertools.chain(), ] for it in iterables: @@ -242,10 +243,44 @@ raises(TypeError, itertools.islice, [], 0, 0, 0, 0) + def test_chain(self): + import itertools + + it = itertools.chain() + raises(StopIteration, it.next) + raises(StopIteration, it.next) + + it = itertools.chain([1, 2, 3]) + for x in [1, 2, 3]: + assert it.next() == x + raises(StopIteration, it.next) + + it = itertools.chain([1, 2, 3], [4], [5, 6]) + for x in [1, 2, 3, 4, 5, 6]: + assert it.next() == x + raises(StopIteration, it.next) + + it = itertools.chain([], [], [1], []) + assert it.next() == 1 + raises(StopIteration, it.next) + + def test_chain_wrongargs(self): + import itertools + + raises(TypeError, itertools.chain, None) + raises(TypeError, itertools.chain, [], None) + + for x in range(10): + args = [()] * x + [None] + [()] * (9 - x) + try: + itertools.chain(*args) + except TypeError, e: + assert str(e) == "chain argument #%d must support iteration" % (x + 1) + def test_docstrings(self): import itertools - assert itertools.__doc__ != "" + assert itertools.__doc__ methods = [ itertools.count, itertools.repeat, @@ -254,10 +289,10 @@ itertools.ifilter, itertools.ifilterfalse, itertools.islice, + itertools.chain, ] for method in methods: - assert method.__doc__ != "" - + assert method.__doc__ def test_subclassing(self): import itertools @@ -270,6 +305,7 @@ (itertools.ifilter, (None, [])), (itertools.ifilterfalse, (None, [])), (itertools.islice, ([], 0)), + (itertools.chain, ()), ] for cls, args in iterables: class A(cls): From hpk at codespeak.net Fri Jul 11 11:14:59 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Fri, 11 Jul 2008 11:14:59 +0200 (CEST) Subject: [pypy-svn] r56439 - in pypy/dist/pypy/lib/xml: . dom etree parsers sax Message-ID: <20080711091459.0A4B4698030@codespeak.net> Author: hpk Date: Fri Jul 11 11:14:56 2008 New Revision: 56439 Added: pypy/dist/pypy/lib/xml/ pypy/dist/pypy/lib/xml/__init__.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/ pypy/dist/pypy/lib/xml/dom/NodeFilter.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/__init__.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/domreg.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/expatbuilder.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/minicompat.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/minidom.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/pulldom.py (contents, props changed) pypy/dist/pypy/lib/xml/dom/xmlbuilder.py (contents, props changed) pypy/dist/pypy/lib/xml/etree/ pypy/dist/pypy/lib/xml/etree/ElementInclude.py (contents, props changed) pypy/dist/pypy/lib/xml/etree/ElementPath.py (contents, props changed) pypy/dist/pypy/lib/xml/etree/ElementTree.py (contents, props changed) pypy/dist/pypy/lib/xml/etree/__init__.py (contents, props changed) pypy/dist/pypy/lib/xml/etree/cElementTree.py (contents, props changed) pypy/dist/pypy/lib/xml/parsers/ pypy/dist/pypy/lib/xml/parsers/__init__.py (contents, props changed) pypy/dist/pypy/lib/xml/parsers/expat.py (contents, props changed) pypy/dist/pypy/lib/xml/sax/ pypy/dist/pypy/lib/xml/sax/__init__.py (contents, props changed) pypy/dist/pypy/lib/xml/sax/_exceptions.py (contents, props changed) pypy/dist/pypy/lib/xml/sax/expatreader.py (contents, props changed) pypy/dist/pypy/lib/xml/sax/handler.py (contents, props changed) pypy/dist/pypy/lib/xml/sax/saxutils.py (contents, props changed) pypy/dist/pypy/lib/xml/sax/xmlreader.py (contents, props changed) Log: adding 2.5's xml version (which includes xml.etree) to our 2.4 based pypy/lib. this helps some more apps to run. Added: pypy/dist/pypy/lib/xml/__init__.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/__init__.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,47 @@ +"""Core XML support for Python. + +This package contains four sub-packages: + +dom -- The W3C Document Object Model. This supports DOM Level 1 + + Namespaces. + +parsers -- Python wrappers for XML parsers (currently only supports Expat). + +sax -- The Simple API for XML, developed by XML-Dev, led by David + Megginson and ported to Python by Lars Marius Garshol. This + supports the SAX 2 API. + +etree -- The ElementTree XML library. This is a subset of the full + ElementTree XML release. + +""" + + +__all__ = ["dom", "parsers", "sax", "etree"] + +# When being checked-out without options, this has the form +# "Revision: x.y " +# When exported using -kv, it is "x.y". +__version__ = "$Revision: 41660 $".split()[-2:][0] + + +_MINIMUM_XMLPLUS_VERSION = (0, 8, 4) + + +try: + import _xmlplus +except ImportError: + pass +else: + try: + v = _xmlplus.version_info + except AttributeError: + # _xmlplus is too old; ignore it + pass + else: + if v >= _MINIMUM_XMLPLUS_VERSION: + import sys + _xmlplus.__path__.extend(__path__) + sys.modules[__name__] = _xmlplus + else: + del v Added: pypy/dist/pypy/lib/xml/dom/NodeFilter.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/NodeFilter.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,27 @@ +# This is the Python mapping for interface NodeFilter from +# DOM2-Traversal-Range. It contains only constants. + +class NodeFilter: + """ + This is the DOM2 NodeFilter interface. It contains only constants. + """ + FILTER_ACCEPT = 1 + FILTER_REJECT = 2 + FILTER_SKIP = 3 + + SHOW_ALL = 0xFFFFFFFFL + SHOW_ELEMENT = 0x00000001 + SHOW_ATTRIBUTE = 0x00000002 + SHOW_TEXT = 0x00000004 + SHOW_CDATA_SECTION = 0x00000008 + SHOW_ENTITY_REFERENCE = 0x00000010 + SHOW_ENTITY = 0x00000020 + SHOW_PROCESSING_INSTRUCTION = 0x00000040 + SHOW_COMMENT = 0x00000080 + SHOW_DOCUMENT = 0x00000100 + SHOW_DOCUMENT_TYPE = 0x00000200 + SHOW_DOCUMENT_FRAGMENT = 0x00000400 + SHOW_NOTATION = 0x00000800 + + def acceptNode(self, node): + raise NotImplementedError Added: pypy/dist/pypy/lib/xml/dom/__init__.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/__init__.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,139 @@ +"""W3C Document Object Model implementation for Python. + +The Python mapping of the Document Object Model is documented in the +Python Library Reference in the section on the xml.dom package. + +This package contains the following modules: + +minidom -- A simple implementation of the Level 1 DOM with namespace + support added (based on the Level 2 specification) and other + minor Level 2 functionality. + +pulldom -- DOM builder supporting on-demand tree-building for selected + subtrees of the document. + +""" + + +class Node: + """Class giving the NodeType constants.""" + + # DOM implementations may use this as a base class for their own + # Node implementations. If they don't, the constants defined here + # should still be used as the canonical definitions as they match + # the values given in the W3C recommendation. Client code can + # safely refer to these values in all tests of Node.nodeType + # values. + + ELEMENT_NODE = 1 + ATTRIBUTE_NODE = 2 + TEXT_NODE = 3 + CDATA_SECTION_NODE = 4 + ENTITY_REFERENCE_NODE = 5 + ENTITY_NODE = 6 + PROCESSING_INSTRUCTION_NODE = 7 + COMMENT_NODE = 8 + DOCUMENT_NODE = 9 + DOCUMENT_TYPE_NODE = 10 + DOCUMENT_FRAGMENT_NODE = 11 + NOTATION_NODE = 12 + + +#ExceptionCode +INDEX_SIZE_ERR = 1 +DOMSTRING_SIZE_ERR = 2 +HIERARCHY_REQUEST_ERR = 3 +WRONG_DOCUMENT_ERR = 4 +INVALID_CHARACTER_ERR = 5 +NO_DATA_ALLOWED_ERR = 6 +NO_MODIFICATION_ALLOWED_ERR = 7 +NOT_FOUND_ERR = 8 +NOT_SUPPORTED_ERR = 9 +INUSE_ATTRIBUTE_ERR = 10 +INVALID_STATE_ERR = 11 +SYNTAX_ERR = 12 +INVALID_MODIFICATION_ERR = 13 +NAMESPACE_ERR = 14 +INVALID_ACCESS_ERR = 15 +VALIDATION_ERR = 16 + + +class DOMException(Exception): + """Abstract base class for DOM exceptions. + Exceptions with specific codes are specializations of this class.""" + + def __init__(self, *args, **kw): + if self.__class__ is DOMException: + raise RuntimeError( + "DOMException should not be instantiated directly") + Exception.__init__(self, *args, **kw) + + def _get_code(self): + return self.code + + +class IndexSizeErr(DOMException): + code = INDEX_SIZE_ERR + +class DomstringSizeErr(DOMException): + code = DOMSTRING_SIZE_ERR + +class HierarchyRequestErr(DOMException): + code = HIERARCHY_REQUEST_ERR + +class WrongDocumentErr(DOMException): + code = WRONG_DOCUMENT_ERR + +class InvalidCharacterErr(DOMException): + code = INVALID_CHARACTER_ERR + +class NoDataAllowedErr(DOMException): + code = NO_DATA_ALLOWED_ERR + +class NoModificationAllowedErr(DOMException): + code = NO_MODIFICATION_ALLOWED_ERR + +class NotFoundErr(DOMException): + code = NOT_FOUND_ERR + +class NotSupportedErr(DOMException): + code = NOT_SUPPORTED_ERR + +class InuseAttributeErr(DOMException): + code = INUSE_ATTRIBUTE_ERR + +class InvalidStateErr(DOMException): + code = INVALID_STATE_ERR + +class SyntaxErr(DOMException): + code = SYNTAX_ERR + +class InvalidModificationErr(DOMException): + code = INVALID_MODIFICATION_ERR + +class NamespaceErr(DOMException): + code = NAMESPACE_ERR + +class InvalidAccessErr(DOMException): + code = INVALID_ACCESS_ERR + +class ValidationErr(DOMException): + code = VALIDATION_ERR + +class UserDataHandler: + """Class giving the operation constants for UserDataHandler.handle().""" + + # Based on DOM Level 3 (WD 9 April 2002) + + NODE_CLONED = 1 + NODE_IMPORTED = 2 + NODE_DELETED = 3 + NODE_RENAMED = 4 + +XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace" +XMLNS_NAMESPACE = "http://www.w3.org/2000/xmlns/" +XHTML_NAMESPACE = "http://www.w3.org/1999/xhtml" +EMPTY_NAMESPACE = None +EMPTY_PREFIX = None + +from domreg import getDOMImplementation,registerDOMImplementation Added: pypy/dist/pypy/lib/xml/dom/domreg.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/domreg.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,99 @@ +"""Registration facilities for DOM. This module should not be used +directly. Instead, the functions getDOMImplementation and +registerDOMImplementation should be imported from xml.dom.""" + +from xml.dom.minicompat import * # isinstance, StringTypes + +# This is a list of well-known implementations. Well-known names +# should be published by posting to xml-sig at python.org, and are +# subsequently recorded in this file. + +well_known_implementations = { + 'minidom':'xml.dom.minidom', + '4DOM': 'xml.dom.DOMImplementation', + } + +# DOM implementations not officially registered should register +# themselves with their + +registered = {} + +def registerDOMImplementation(name, factory): + """registerDOMImplementation(name, factory) + + Register the factory function with the name. The factory function + should return an object which implements the DOMImplementation + interface. The factory function can either return the same object, + or a new one (e.g. if that implementation supports some + customization).""" + + registered[name] = factory + +def _good_enough(dom, features): + "_good_enough(dom, features) -> Return 1 if the dom offers the features" + for f,v in features: + if not dom.hasFeature(f,v): + return 0 + return 1 + +def getDOMImplementation(name = None, features = ()): + """getDOMImplementation(name = None, features = ()) -> DOM implementation. + + Return a suitable DOM implementation. The name is either + well-known, the module name of a DOM implementation, or None. If + it is not None, imports the corresponding module and returns + DOMImplementation object if the import succeeds. + + If name is not given, consider the available implementations to + find one with the required feature set. If no implementation can + be found, raise an ImportError. The features list must be a sequence + of (feature, version) pairs which are passed to hasFeature.""" + + import os + creator = None + mod = well_known_implementations.get(name) + if mod: + mod = __import__(mod, {}, {}, ['getDOMImplementation']) + return mod.getDOMImplementation() + elif name: + return registered[name]() + elif os.environ.has_key("PYTHON_DOM"): + return getDOMImplementation(name = os.environ["PYTHON_DOM"]) + + # User did not specify a name, try implementations in arbitrary + # order, returning the one that has the required features + if isinstance(features, StringTypes): + features = _parse_feature_string(features) + for creator in registered.values(): + dom = creator() + if _good_enough(dom, features): + return dom + + for creator in well_known_implementations.keys(): + try: + dom = getDOMImplementation(name = creator) + except StandardError: # typically ImportError, or AttributeError + continue + if _good_enough(dom, features): + return dom + + raise ImportError,"no suitable DOM implementation found" + +def _parse_feature_string(s): + features = [] + parts = s.split() + i = 0 + length = len(parts) + while i < length: + feature = parts[i] + if feature[0] in "0123456789": + raise ValueError, "bad feature name: %r" % (feature,) + i = i + 1 + version = None + if i < length: + v = parts[i] + if v[0] in "0123456789": + i = i + 1 + version = v + features.append((feature, version)) + return tuple(features) Added: pypy/dist/pypy/lib/xml/dom/expatbuilder.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/expatbuilder.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,983 @@ +"""Facility to use the Expat parser to load a minidom instance +from a string or file. + +This avoids all the overhead of SAX and pulldom to gain performance. +""" + +# Warning! +# +# This module is tightly bound to the implementation details of the +# minidom DOM and can't be used with other DOM implementations. This +# is due, in part, to a lack of appropriate methods in the DOM (there is +# no way to create Entity and Notation nodes via the DOM Level 2 +# interface), and for performance. The later is the cause of some fairly +# cryptic code. +# +# Performance hacks: +# +# - .character_data_handler() has an extra case in which continuing +# data is appended to an existing Text node; this can be a +# speedup since pyexpat can break up character data into multiple +# callbacks even though we set the buffer_text attribute on the +# parser. This also gives us the advantage that we don't need a +# separate normalization pass. +# +# - Determining that a node exists is done using an identity comparison +# with None rather than a truth test; this avoids searching for and +# calling any methods on the node object if it exists. (A rather +# nice speedup is achieved this way as well!) + +from xml.dom import xmlbuilder, minidom, Node +from xml.dom import EMPTY_NAMESPACE, EMPTY_PREFIX, XMLNS_NAMESPACE +from xml.parsers import expat +from xml.dom.minidom import _append_child, _set_attribute_node +from xml.dom.NodeFilter import NodeFilter + +from xml.dom.minicompat import * + +TEXT_NODE = Node.TEXT_NODE +CDATA_SECTION_NODE = Node.CDATA_SECTION_NODE +DOCUMENT_NODE = Node.DOCUMENT_NODE + +FILTER_ACCEPT = xmlbuilder.DOMBuilderFilter.FILTER_ACCEPT +FILTER_REJECT = xmlbuilder.DOMBuilderFilter.FILTER_REJECT +FILTER_SKIP = xmlbuilder.DOMBuilderFilter.FILTER_SKIP +FILTER_INTERRUPT = xmlbuilder.DOMBuilderFilter.FILTER_INTERRUPT + +theDOMImplementation = minidom.getDOMImplementation() + +# Expat typename -> TypeInfo +_typeinfo_map = { + "CDATA": minidom.TypeInfo(None, "cdata"), + "ENUM": minidom.TypeInfo(None, "enumeration"), + "ENTITY": minidom.TypeInfo(None, "entity"), + "ENTITIES": minidom.TypeInfo(None, "entities"), + "ID": minidom.TypeInfo(None, "id"), + "IDREF": minidom.TypeInfo(None, "idref"), + "IDREFS": minidom.TypeInfo(None, "idrefs"), + "NMTOKEN": minidom.TypeInfo(None, "nmtoken"), + "NMTOKENS": minidom.TypeInfo(None, "nmtokens"), + } + +class ElementInfo(object): + __slots__ = '_attr_info', '_model', 'tagName' + + def __init__(self, tagName, model=None): + self.tagName = tagName + self._attr_info = [] + self._model = model + + def __getstate__(self): + return self._attr_info, self._model, self.tagName + + def __setstate__(self, state): + self._attr_info, self._model, self.tagName = state + + def getAttributeType(self, aname): + for info in self._attr_info: + if info[1] == aname: + t = info[-2] + if t[0] == "(": + return _typeinfo_map["ENUM"] + else: + return _typeinfo_map[info[-2]] + return minidom._no_type + + def getAttributeTypeNS(self, namespaceURI, localName): + return minidom._no_type + + def isElementContent(self): + if self._model: + type = self._model[0] + return type not in (expat.model.XML_CTYPE_ANY, + expat.model.XML_CTYPE_MIXED) + else: + return False + + def isEmpty(self): + if self._model: + return self._model[0] == expat.model.XML_CTYPE_EMPTY + else: + return False + + def isId(self, aname): + for info in self._attr_info: + if info[1] == aname: + return info[-2] == "ID" + return False + + def isIdNS(self, euri, ename, auri, aname): + # not sure this is meaningful + return self.isId((auri, aname)) + +def _intern(builder, s): + return builder._intern_setdefault(s, s) + +def _parse_ns_name(builder, name): + assert ' ' in name + parts = name.split(' ') + intern = builder._intern_setdefault + if len(parts) == 3: + uri, localname, prefix = parts + prefix = intern(prefix, prefix) + qname = "%s:%s" % (prefix, localname) + qname = intern(qname, qname) + localname = intern(localname, localname) + else: + uri, localname = parts + prefix = EMPTY_PREFIX + qname = localname = intern(localname, localname) + return intern(uri, uri), localname, prefix, qname + + +class ExpatBuilder: + """Document builder that uses Expat to build a ParsedXML.DOM document + instance.""" + + def __init__(self, options=None): + if options is None: + options = xmlbuilder.Options() + self._options = options + if self._options.filter is not None: + self._filter = FilterVisibilityController(self._options.filter) + else: + self._filter = None + # This *really* doesn't do anything in this case, so + # override it with something fast & minimal. + self._finish_start_element = id + self._parser = None + self.reset() + + def createParser(self): + """Create a new parser object.""" + return expat.ParserCreate() + + def getParser(self): + """Return the parser object, creating a new one if needed.""" + if not self._parser: + self._parser = self.createParser() + self._intern_setdefault = self._parser.intern.setdefault + self._parser.buffer_text = True + self._parser.ordered_attributes = True + self._parser.specified_attributes = True + self.install(self._parser) + return self._parser + + def reset(self): + """Free all data structures used during DOM construction.""" + self.document = theDOMImplementation.createDocument( + EMPTY_NAMESPACE, None, None) + self.curNode = self.document + self._elem_info = self.document._elem_info + self._cdata = False + + def install(self, parser): + """Install the callbacks needed to build the DOM into the parser.""" + # This creates circular references! + parser.StartDoctypeDeclHandler = self.start_doctype_decl_handler + parser.StartElementHandler = self.first_element_handler + parser.EndElementHandler = self.end_element_handler + parser.ProcessingInstructionHandler = self.pi_handler + if self._options.entities: + parser.EntityDeclHandler = self.entity_decl_handler + parser.NotationDeclHandler = self.notation_decl_handler + if self._options.comments: + parser.CommentHandler = self.comment_handler + if self._options.cdata_sections: + parser.StartCdataSectionHandler = self.start_cdata_section_handler + parser.EndCdataSectionHandler = self.end_cdata_section_handler + parser.CharacterDataHandler = self.character_data_handler_cdata + else: + parser.CharacterDataHandler = self.character_data_handler + parser.ExternalEntityRefHandler = self.external_entity_ref_handler + parser.XmlDeclHandler = self.xml_decl_handler + parser.ElementDeclHandler = self.element_decl_handler + parser.AttlistDeclHandler = self.attlist_decl_handler + + def parseFile(self, file): + """Parse a document from a file object, returning the document + node.""" + parser = self.getParser() + first_buffer = True + try: + while 1: + buffer = file.read(16*1024) + if not buffer: + break + parser.Parse(buffer, 0) + if first_buffer and self.document.documentElement: + self._setup_subset(buffer) + first_buffer = False + parser.Parse("", True) + except ParseEscape: + pass + doc = self.document + self.reset() + self._parser = None + return doc + + def parseString(self, string): + """Parse a document from a string, returning the document node.""" + parser = self.getParser() + try: + parser.Parse(string, True) + self._setup_subset(string) + except ParseEscape: + pass + doc = self.document + self.reset() + self._parser = None + return doc + + def _setup_subset(self, buffer): + """Load the internal subset if there might be one.""" + if self.document.doctype: + extractor = InternalSubsetExtractor() + extractor.parseString(buffer) + subset = extractor.getSubset() + self.document.doctype.internalSubset = subset + + def start_doctype_decl_handler(self, doctypeName, systemId, publicId, + has_internal_subset): + doctype = self.document.implementation.createDocumentType( + doctypeName, publicId, systemId) + doctype.ownerDocument = self.document + self.document.childNodes.append(doctype) + self.document.doctype = doctype + if self._filter and self._filter.acceptNode(doctype) == FILTER_REJECT: + self.document.doctype = None + del self.document.childNodes[-1] + doctype = None + self._parser.EntityDeclHandler = None + self._parser.NotationDeclHandler = None + if has_internal_subset: + if doctype is not None: + doctype.entities._seq = [] + doctype.notations._seq = [] + self._parser.CommentHandler = None + self._parser.ProcessingInstructionHandler = None + self._parser.EndDoctypeDeclHandler = self.end_doctype_decl_handler + + def end_doctype_decl_handler(self): + if self._options.comments: + self._parser.CommentHandler = self.comment_handler + self._parser.ProcessingInstructionHandler = self.pi_handler + if not (self._elem_info or self._filter): + self._finish_end_element = id + + def pi_handler(self, target, data): + node = self.document.createProcessingInstruction(target, data) + _append_child(self.curNode, node) + if self._filter and self._filter.acceptNode(node) == FILTER_REJECT: + self.curNode.removeChild(node) + + def character_data_handler_cdata(self, data): + childNodes = self.curNode.childNodes + if self._cdata: + if ( self._cdata_continue + and childNodes[-1].nodeType == CDATA_SECTION_NODE): + childNodes[-1].appendData(data) + return + node = self.document.createCDATASection(data) + self._cdata_continue = True + elif childNodes and childNodes[-1].nodeType == TEXT_NODE: + node = childNodes[-1] + value = node.data + data + d = node.__dict__ + d['data'] = d['nodeValue'] = value + return + else: + node = minidom.Text() + d = node.__dict__ + d['data'] = d['nodeValue'] = data + d['ownerDocument'] = self.document + _append_child(self.curNode, node) + + def character_data_handler(self, data): + childNodes = self.curNode.childNodes + if childNodes and childNodes[-1].nodeType == TEXT_NODE: + node = childNodes[-1] + d = node.__dict__ + d['data'] = d['nodeValue'] = node.data + data + return + node = minidom.Text() + d = node.__dict__ + d['data'] = d['nodeValue'] = node.data + data + d['ownerDocument'] = self.document + _append_child(self.curNode, node) + + def entity_decl_handler(self, entityName, is_parameter_entity, value, + base, systemId, publicId, notationName): + if is_parameter_entity: + # we don't care about parameter entities for the DOM + return + if not self._options.entities: + return + node = self.document._create_entity(entityName, publicId, + systemId, notationName) + if value is not None: + # internal entity + # node *should* be readonly, but we'll cheat + child = self.document.createTextNode(value) + node.childNodes.append(child) + self.document.doctype.entities._seq.append(node) + if self._filter and self._filter.acceptNode(node) == FILTER_REJECT: + del self.document.doctype.entities._seq[-1] + + def notation_decl_handler(self, notationName, base, systemId, publicId): + node = self.document._create_notation(notationName, publicId, systemId) + self.document.doctype.notations._seq.append(node) + if self._filter and self._filter.acceptNode(node) == FILTER_ACCEPT: + del self.document.doctype.notations._seq[-1] + + def comment_handler(self, data): + node = self.document.createComment(data) + _append_child(self.curNode, node) + if self._filter and self._filter.acceptNode(node) == FILTER_REJECT: + self.curNode.removeChild(node) + + def start_cdata_section_handler(self): + self._cdata = True + self._cdata_continue = False + + def end_cdata_section_handler(self): + self._cdata = False + self._cdata_continue = False + + def external_entity_ref_handler(self, context, base, systemId, publicId): + return 1 + + def first_element_handler(self, name, attributes): + if self._filter is None and not self._elem_info: + self._finish_end_element = id + self.getParser().StartElementHandler = self.start_element_handler + self.start_element_handler(name, attributes) + + def start_element_handler(self, name, attributes): + node = self.document.createElement(name) + _append_child(self.curNode, node) + self.curNode = node + + if attributes: + for i in range(0, len(attributes), 2): + a = minidom.Attr(attributes[i], EMPTY_NAMESPACE, + None, EMPTY_PREFIX) + value = attributes[i+1] + d = a.childNodes[0].__dict__ + d['data'] = d['nodeValue'] = value + d = a.__dict__ + d['value'] = d['nodeValue'] = value + d['ownerDocument'] = self.document + _set_attribute_node(node, a) + + if node is not self.document.documentElement: + self._finish_start_element(node) + + def _finish_start_element(self, node): + if self._filter: + # To be general, we'd have to call isSameNode(), but this + # is sufficient for minidom: + if node is self.document.documentElement: + return + filt = self._filter.startContainer(node) + if filt == FILTER_REJECT: + # ignore this node & all descendents + Rejecter(self) + elif filt == FILTER_SKIP: + # ignore this node, but make it's children become + # children of the parent node + Skipper(self) + else: + return + self.curNode = node.parentNode + node.parentNode.removeChild(node) + node.unlink() + + # If this ever changes, Namespaces.end_element_handler() needs to + # be changed to match. + # + def end_element_handler(self, name): + curNode = self.curNode + self.curNode = curNode.parentNode + self._finish_end_element(curNode) + + def _finish_end_element(self, curNode): + info = self._elem_info.get(curNode.tagName) + if info: + self._handle_white_text_nodes(curNode, info) + if self._filter: + if curNode is self.document.documentElement: + return + if self._filter.acceptNode(curNode) == FILTER_REJECT: + self.curNode.removeChild(curNode) + curNode.unlink() + + def _handle_white_text_nodes(self, node, info): + if (self._options.whitespace_in_element_content + or not info.isElementContent()): + return + + # We have element type information and should remove ignorable + # whitespace; identify for text nodes which contain only + # whitespace. + L = [] + for child in node.childNodes: + if child.nodeType == TEXT_NODE and not child.data.strip(): + L.append(child) + + # Remove ignorable whitespace from the tree. + for child in L: + node.removeChild(child) + + def element_decl_handler(self, name, model): + info = self._elem_info.get(name) + if info is None: + self._elem_info[name] = ElementInfo(name, model) + else: + assert info._model is None + info._model = model + + def attlist_decl_handler(self, elem, name, type, default, required): + info = self._elem_info.get(elem) + if info is None: + info = ElementInfo(elem) + self._elem_info[elem] = info + info._attr_info.append( + [None, name, None, None, default, 0, type, required]) + + def xml_decl_handler(self, version, encoding, standalone): + self.document.version = version + self.document.encoding = encoding + # This is still a little ugly, thanks to the pyexpat API. ;-( + if standalone >= 0: + if standalone: + self.document.standalone = True + else: + self.document.standalone = False + + +# Don't include FILTER_INTERRUPT, since that's checked separately +# where allowed. +_ALLOWED_FILTER_RETURNS = (FILTER_ACCEPT, FILTER_REJECT, FILTER_SKIP) + +class FilterVisibilityController(object): + """Wrapper around a DOMBuilderFilter which implements the checks + to make the whatToShow filter attribute work.""" + + __slots__ = 'filter', + + def __init__(self, filter): + self.filter = filter + + def startContainer(self, node): + mask = self._nodetype_mask[node.nodeType] + if self.filter.whatToShow & mask: + val = self.filter.startContainer(node) + if val == FILTER_INTERRUPT: + raise ParseEscape + if val not in _ALLOWED_FILTER_RETURNS: + raise ValueError, \ + "startContainer() returned illegal value: " + repr(val) + return val + else: + return FILTER_ACCEPT + + def acceptNode(self, node): + mask = self._nodetype_mask[node.nodeType] + if self.filter.whatToShow & mask: + val = self.filter.acceptNode(node) + if val == FILTER_INTERRUPT: + raise ParseEscape + if val == FILTER_SKIP: + # move all child nodes to the parent, and remove this node + parent = node.parentNode + for child in node.childNodes[:]: + parent.appendChild(child) + # node is handled by the caller + return FILTER_REJECT + if val not in _ALLOWED_FILTER_RETURNS: + raise ValueError, \ + "acceptNode() returned illegal value: " + repr(val) + return val + else: + return FILTER_ACCEPT + + _nodetype_mask = { + Node.ELEMENT_NODE: NodeFilter.SHOW_ELEMENT, + Node.ATTRIBUTE_NODE: NodeFilter.SHOW_ATTRIBUTE, + Node.TEXT_NODE: NodeFilter.SHOW_TEXT, + Node.CDATA_SECTION_NODE: NodeFilter.SHOW_CDATA_SECTION, + Node.ENTITY_REFERENCE_NODE: NodeFilter.SHOW_ENTITY_REFERENCE, + Node.ENTITY_NODE: NodeFilter.SHOW_ENTITY, + Node.PROCESSING_INSTRUCTION_NODE: NodeFilter.SHOW_PROCESSING_INSTRUCTION, + Node.COMMENT_NODE: NodeFilter.SHOW_COMMENT, + Node.DOCUMENT_NODE: NodeFilter.SHOW_DOCUMENT, + Node.DOCUMENT_TYPE_NODE: NodeFilter.SHOW_DOCUMENT_TYPE, + Node.DOCUMENT_FRAGMENT_NODE: NodeFilter.SHOW_DOCUMENT_FRAGMENT, + Node.NOTATION_NODE: NodeFilter.SHOW_NOTATION, + } + + +class FilterCrutch(object): + __slots__ = '_builder', '_level', '_old_start', '_old_end' + + def __init__(self, builder): + self._level = 0 + self._builder = builder + parser = builder._parser + self._old_start = parser.StartElementHandler + self._old_end = parser.EndElementHandler + parser.StartElementHandler = self.start_element_handler + parser.EndElementHandler = self.end_element_handler + +class Rejecter(FilterCrutch): + __slots__ = () + + def __init__(self, builder): + FilterCrutch.__init__(self, builder) + parser = builder._parser + for name in ("ProcessingInstructionHandler", + "CommentHandler", + "CharacterDataHandler", + "StartCdataSectionHandler", + "EndCdataSectionHandler", + "ExternalEntityRefHandler", + ): + setattr(parser, name, None) + + def start_element_handler(self, *args): + self._level = self._level + 1 + + def end_element_handler(self, *args): + if self._level == 0: + # restore the old handlers + parser = self._builder._parser + self._builder.install(parser) + parser.StartElementHandler = self._old_start + parser.EndElementHandler = self._old_end + else: + self._level = self._level - 1 + +class Skipper(FilterCrutch): + __slots__ = () + + def start_element_handler(self, *args): + node = self._builder.curNode + self._old_start(*args) + if self._builder.curNode is not node: + self._level = self._level + 1 + + def end_element_handler(self, *args): + if self._level == 0: + # We're popping back out of the node we're skipping, so we + # shouldn't need to do anything but reset the handlers. + self._builder._parser.StartElementHandler = self._old_start + self._builder._parser.EndElementHandler = self._old_end + self._builder = None + else: + self._level = self._level - 1 + self._old_end(*args) + + +# framework document used by the fragment builder. +# Takes a string for the doctype, subset string, and namespace attrs string. + +_FRAGMENT_BUILDER_INTERNAL_SYSTEM_ID = \ + "http://xml.python.org/entities/fragment-builder/internal" + +_FRAGMENT_BUILDER_TEMPLATE = ( + '''\ + +%%s +]> +&fragment-builder-internal;''' + % _FRAGMENT_BUILDER_INTERNAL_SYSTEM_ID) + + +class FragmentBuilder(ExpatBuilder): + """Builder which constructs document fragments given XML source + text and a context node. + + The context node is expected to provide information about the + namespace declarations which are in scope at the start of the + fragment. + """ + + def __init__(self, context, options=None): + if context.nodeType == DOCUMENT_NODE: + self.originalDocument = context + self.context = context + else: + self.originalDocument = context.ownerDocument + self.context = context + ExpatBuilder.__init__(self, options) + + def reset(self): + ExpatBuilder.reset(self) + self.fragment = None + + def parseFile(self, file): + """Parse a document fragment from a file object, returning the + fragment node.""" + return self.parseString(file.read()) + + def parseString(self, string): + """Parse a document fragment from a string, returning the + fragment node.""" + self._source = string + parser = self.getParser() + doctype = self.originalDocument.doctype + ident = "" + if doctype: + subset = doctype.internalSubset or self._getDeclarations() + if doctype.publicId: + ident = ('PUBLIC "%s" "%s"' + % (doctype.publicId, doctype.systemId)) + elif doctype.systemId: + ident = 'SYSTEM "%s"' % doctype.systemId + else: + subset = "" + nsattrs = self._getNSattrs() # get ns decls from node's ancestors + document = _FRAGMENT_BUILDER_TEMPLATE % (ident, subset, nsattrs) + try: + parser.Parse(document, 1) + except: + self.reset() + raise + fragment = self.fragment + self.reset() +## self._parser = None + return fragment + + def _getDeclarations(self): + """Re-create the internal subset from the DocumentType node. + + This is only needed if we don't already have the + internalSubset as a string. + """ + doctype = self.context.ownerDocument.doctype + s = "" + if doctype: + for i in range(doctype.notations.length): + notation = doctype.notations.item(i) + if s: + s = s + "\n " + s = "%s' \ + % (s, notation.publicId, notation.systemId) + else: + s = '%s SYSTEM "%s">' % (s, notation.systemId) + for i in range(doctype.entities.length): + entity = doctype.entities.item(i) + if s: + s = s + "\n " + s = "%s" + return s + + def _getNSattrs(self): + return "" + + def external_entity_ref_handler(self, context, base, systemId, publicId): + if systemId == _FRAGMENT_BUILDER_INTERNAL_SYSTEM_ID: + # this entref is the one that we made to put the subtree + # in; all of our given input is parsed in here. + old_document = self.document + old_cur_node = self.curNode + parser = self._parser.ExternalEntityParserCreate(context) + # put the real document back, parse into the fragment to return + self.document = self.originalDocument + self.fragment = self.document.createDocumentFragment() + self.curNode = self.fragment + try: + parser.Parse(self._source, 1) + finally: + self.curNode = old_cur_node + self.document = old_document + self._source = None + return -1 + else: + return ExpatBuilder.external_entity_ref_handler( + self, context, base, systemId, publicId) + + +class Namespaces: + """Mix-in class for builders; adds support for namespaces.""" + + def _initNamespaces(self): + # list of (prefix, uri) ns declarations. Namespace attrs are + # constructed from this and added to the element's attrs. + self._ns_ordered_prefixes = [] + + def createParser(self): + """Create a new namespace-handling parser.""" + parser = expat.ParserCreate(namespace_separator=" ") + parser.namespace_prefixes = True + return parser + + def install(self, parser): + """Insert the namespace-handlers onto the parser.""" + ExpatBuilder.install(self, parser) + if self._options.namespace_declarations: + parser.StartNamespaceDeclHandler = ( + self.start_namespace_decl_handler) + + def start_namespace_decl_handler(self, prefix, uri): + """Push this namespace declaration on our storage.""" + self._ns_ordered_prefixes.append((prefix, uri)) + + def start_element_handler(self, name, attributes): + if ' ' in name: + uri, localname, prefix, qname = _parse_ns_name(self, name) + else: + uri = EMPTY_NAMESPACE + qname = name + localname = None + prefix = EMPTY_PREFIX + node = minidom.Element(qname, uri, prefix, localname) + node.ownerDocument = self.document + _append_child(self.curNode, node) + self.curNode = node + + if self._ns_ordered_prefixes: + for prefix, uri in self._ns_ordered_prefixes: + if prefix: + a = minidom.Attr(_intern(self, 'xmlns:' + prefix), + XMLNS_NAMESPACE, prefix, "xmlns") + else: + a = minidom.Attr("xmlns", XMLNS_NAMESPACE, + "xmlns", EMPTY_PREFIX) + d = a.childNodes[0].__dict__ + d['data'] = d['nodeValue'] = uri + d = a.__dict__ + d['value'] = d['nodeValue'] = uri + d['ownerDocument'] = self.document + _set_attribute_node(node, a) + del self._ns_ordered_prefixes[:] + + if attributes: + _attrs = node._attrs + _attrsNS = node._attrsNS + for i in range(0, len(attributes), 2): + aname = attributes[i] + value = attributes[i+1] + if ' ' in aname: + uri, localname, prefix, qname = _parse_ns_name(self, aname) + a = minidom.Attr(qname, uri, localname, prefix) + _attrs[qname] = a + _attrsNS[(uri, localname)] = a + else: + a = minidom.Attr(aname, EMPTY_NAMESPACE, + aname, EMPTY_PREFIX) + _attrs[aname] = a + _attrsNS[(EMPTY_NAMESPACE, aname)] = a + d = a.childNodes[0].__dict__ + d['data'] = d['nodeValue'] = value + d = a.__dict__ + d['ownerDocument'] = self.document + d['value'] = d['nodeValue'] = value + d['ownerElement'] = node + + if __debug__: + # This only adds some asserts to the original + # end_element_handler(), so we only define this when -O is not + # used. If changing one, be sure to check the other to see if + # it needs to be changed as well. + # + def end_element_handler(self, name): + curNode = self.curNode + if ' ' in name: + uri, localname, prefix, qname = _parse_ns_name(self, name) + assert (curNode.namespaceURI == uri + and curNode.localName == localname + and curNode.prefix == prefix), \ + "element stack messed up! (namespace)" + else: + assert curNode.nodeName == name, \ + "element stack messed up - bad nodeName" + assert curNode.namespaceURI == EMPTY_NAMESPACE, \ + "element stack messed up - bad namespaceURI" + self.curNode = curNode.parentNode + self._finish_end_element(curNode) + + +class ExpatBuilderNS(Namespaces, ExpatBuilder): + """Document builder that supports namespaces.""" + + def reset(self): + ExpatBuilder.reset(self) + self._initNamespaces() + + +class FragmentBuilderNS(Namespaces, FragmentBuilder): + """Fragment builder that supports namespaces.""" + + def reset(self): + FragmentBuilder.reset(self) + self._initNamespaces() + + def _getNSattrs(self): + """Return string of namespace attributes from this element and + ancestors.""" + # XXX This needs to be re-written to walk the ancestors of the + # context to build up the namespace information from + # declarations, elements, and attributes found in context. + # Otherwise we have to store a bunch more data on the DOM + # (though that *might* be more reliable -- not clear). + attrs = "" + context = self.context + L = [] + while context: + if hasattr(context, '_ns_prefix_uri'): + for prefix, uri in context._ns_prefix_uri.items(): + # add every new NS decl from context to L and attrs string + if prefix in L: + continue + L.append(prefix) + if prefix: + declname = "xmlns:" + prefix + else: + declname = "xmlns" + if attrs: + attrs = "%s\n %s='%s'" % (attrs, declname, uri) + else: + attrs = " %s='%s'" % (declname, uri) + context = context.parentNode + return attrs + + +class ParseEscape(Exception): + """Exception raised to short-circuit parsing in InternalSubsetExtractor.""" + pass + +class InternalSubsetExtractor(ExpatBuilder): + """XML processor which can rip out the internal document type subset.""" + + subset = None + + def getSubset(self): + """Return the internal subset as a string.""" + return self.subset + + def parseFile(self, file): + try: + ExpatBuilder.parseFile(self, file) + except ParseEscape: + pass + + def parseString(self, string): + try: + ExpatBuilder.parseString(self, string) + except ParseEscape: + pass + + def install(self, parser): + parser.StartDoctypeDeclHandler = self.start_doctype_decl_handler + parser.StartElementHandler = self.start_element_handler + + def start_doctype_decl_handler(self, name, publicId, systemId, + has_internal_subset): + if has_internal_subset: + parser = self.getParser() + self.subset = [] + parser.DefaultHandler = self.subset.append + parser.EndDoctypeDeclHandler = self.end_doctype_decl_handler + else: + raise ParseEscape() + + def end_doctype_decl_handler(self): + s = ''.join(self.subset).replace('\r\n', '\n').replace('\r', '\n') + self.subset = s + raise ParseEscape() + + def start_element_handler(self, name, attrs): + raise ParseEscape() + + +def parse(file, namespaces=True): + """Parse a document, returning the resulting Document node. + + 'file' may be either a file name or an open file object. + """ + if namespaces: + builder = ExpatBuilderNS() + else: + builder = ExpatBuilder() + + if isinstance(file, StringTypes): + fp = open(file, 'rb') + try: + result = builder.parseFile(fp) + finally: + fp.close() + else: + result = builder.parseFile(file) + return result + + +def parseString(string, namespaces=True): + """Parse a document from a string, returning the resulting + Document node. + """ + if namespaces: + builder = ExpatBuilderNS() + else: + builder = ExpatBuilder() + return builder.parseString(string) + + +def parseFragment(file, context, namespaces=True): + """Parse a fragment of a document, given the context from which it + was originally extracted. context should be the parent of the + node(s) which are in the fragment. + + 'file' may be either a file name or an open file object. + """ + if namespaces: + builder = FragmentBuilderNS(context) + else: + builder = FragmentBuilder(context) + + if isinstance(file, StringTypes): + fp = open(file, 'rb') + try: + result = builder.parseFile(fp) + finally: + fp.close() + else: + result = builder.parseFile(file) + return result + + +def parseFragmentString(string, context, namespaces=True): + """Parse a fragment of a document from a string, given the context + from which it was originally extracted. context should be the + parent of the node(s) which are in the fragment. + """ + if namespaces: + builder = FragmentBuilderNS(context) + else: + builder = FragmentBuilder(context) + return builder.parseString(string) + + +def makeBuilder(options): + """Create a builder based on an Options object.""" + if options.namespaces: + return ExpatBuilderNS(options) + else: + return ExpatBuilder(options) Added: pypy/dist/pypy/lib/xml/dom/minicompat.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/minicompat.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,110 @@ +"""Python version compatibility support for minidom.""" + +# This module should only be imported using "import *". +# +# The following names are defined: +# +# NodeList -- lightest possible NodeList implementation +# +# EmptyNodeList -- lightest possible NodeList that is guarateed to +# remain empty (immutable) +# +# StringTypes -- tuple of defined string types +# +# defproperty -- function used in conjunction with GetattrMagic; +# using these together is needed to make them work +# as efficiently as possible in both Python 2.2+ +# and older versions. For example: +# +# class MyClass(GetattrMagic): +# def _get_myattr(self): +# return something +# +# defproperty(MyClass, "myattr", +# "return some value") +# +# For Python 2.2 and newer, this will construct a +# property object on the class, which avoids +# needing to override __getattr__(). It will only +# work for read-only attributes. +# +# For older versions of Python, inheriting from +# GetattrMagic will use the traditional +# __getattr__() hackery to achieve the same effect, +# but less efficiently. +# +# defproperty() should be used for each version of +# the relevant _get_() function. + +__all__ = ["NodeList", "EmptyNodeList", "StringTypes", "defproperty"] + +import xml.dom + +try: + unicode +except NameError: + StringTypes = type(''), +else: + StringTypes = type(''), type(unicode('')) + + +class NodeList(list): + __slots__ = () + + def item(self, index): + if 0 <= index < len(self): + return self[index] + + def _get_length(self): + return len(self) + + def _set_length(self, value): + raise xml.dom.NoModificationAllowedErr( + "attempt to modify read-only attribute 'length'") + + length = property(_get_length, _set_length, + doc="The number of nodes in the NodeList.") + + def __getstate__(self): + return list(self) + + def __setstate__(self, state): + self[:] = state + + +class EmptyNodeList(tuple): + __slots__ = () + + def __add__(self, other): + NL = NodeList() + NL.extend(other) + return NL + + def __radd__(self, other): + NL = NodeList() + NL.extend(other) + return NL + + def item(self, index): + return None + + def _get_length(self): + return 0 + + def _set_length(self, value): + raise xml.dom.NoModificationAllowedErr( + "attempt to modify read-only attribute 'length'") + + length = property(_get_length, _set_length, + doc="The number of nodes in the NodeList.") + + +def defproperty(klass, name, doc): + get = getattr(klass, ("_get_" + name)).im_func + def set(self, value, name=name): + raise xml.dom.NoModificationAllowedErr( + "attempt to modify read-only attribute " + repr(name)) + assert not hasattr(klass, "_set_" + name), \ + "expected not to find _set_" + name + prop = property(get, set, doc=doc) + setattr(klass, name, prop) Added: pypy/dist/pypy/lib/xml/dom/minidom.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/minidom.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,1936 @@ +"""\ +minidom.py -- a lightweight DOM implementation. + +parse("foo.xml") + +parseString("") + +Todo: +===== + * convenience methods for getting elements and text. + * more testing + * bring some of the writer and linearizer code into conformance with this + interface + * SAX 2 namespaces +""" + +import xml.dom + +from xml.dom import EMPTY_NAMESPACE, EMPTY_PREFIX, XMLNS_NAMESPACE, domreg +from xml.dom.minicompat import * +from xml.dom.xmlbuilder import DOMImplementationLS, DocumentLS + +# This is used by the ID-cache invalidation checks; the list isn't +# actually complete, since the nodes being checked will never be the +# DOCUMENT_NODE or DOCUMENT_FRAGMENT_NODE. (The node being checked is +# the node being added or removed, not the node being modified.) +# +_nodeTypes_with_children = (xml.dom.Node.ELEMENT_NODE, + xml.dom.Node.ENTITY_REFERENCE_NODE) + + +class Node(xml.dom.Node): + namespaceURI = None # this is non-null only for elements and attributes + parentNode = None + ownerDocument = None + nextSibling = None + previousSibling = None + + prefix = EMPTY_PREFIX # non-null only for NS elements and attributes + + def __nonzero__(self): + return True + + def toxml(self, encoding = None): + return self.toprettyxml("", "", encoding) + + def toprettyxml(self, indent="\t", newl="\n", encoding = None): + # indent = the indentation string to prepend, per level + # newl = the newline string to append + writer = _get_StringIO() + if encoding is not None: + import codecs + # Can't use codecs.getwriter to preserve 2.0 compatibility + writer = codecs.lookup(encoding)[3](writer) + if self.nodeType == Node.DOCUMENT_NODE: + # Can pass encoding only to document, to put it into XML header + self.writexml(writer, "", indent, newl, encoding) + else: + self.writexml(writer, "", indent, newl) + return writer.getvalue() + + def hasChildNodes(self): + if self.childNodes: + return True + else: + return False + + def _get_childNodes(self): + return self.childNodes + + def _get_firstChild(self): + if self.childNodes: + return self.childNodes[0] + + def _get_lastChild(self): + if self.childNodes: + return self.childNodes[-1] + + def insertBefore(self, newChild, refChild): + if newChild.nodeType == self.DOCUMENT_FRAGMENT_NODE: + for c in tuple(newChild.childNodes): + self.insertBefore(c, refChild) + ### The DOM does not clearly specify what to return in this case + return newChild + if newChild.nodeType not in self._child_node_types: + raise xml.dom.HierarchyRequestErr( + "%s cannot be child of %s" % (repr(newChild), repr(self))) + if newChild.parentNode is not None: + newChild.parentNode.removeChild(newChild) + if refChild is None: + self.appendChild(newChild) + else: + try: + index = self.childNodes.index(refChild) + except ValueError: + raise xml.dom.NotFoundErr() + if newChild.nodeType in _nodeTypes_with_children: + _clear_id_cache(self) + self.childNodes.insert(index, newChild) + newChild.nextSibling = refChild + refChild.previousSibling = newChild + if index: + node = self.childNodes[index-1] + node.nextSibling = newChild + newChild.previousSibling = node + else: + newChild.previousSibling = None + newChild.parentNode = self + return newChild + + def appendChild(self, node): + if node.nodeType == self.DOCUMENT_FRAGMENT_NODE: + for c in tuple(node.childNodes): + self.appendChild(c) + ### The DOM does not clearly specify what to return in this case + return node + if node.nodeType not in self._child_node_types: + raise xml.dom.HierarchyRequestErr( + "%s cannot be child of %s" % (repr(node), repr(self))) + elif node.nodeType in _nodeTypes_with_children: + _clear_id_cache(self) + if node.parentNode is not None: + node.parentNode.removeChild(node) + _append_child(self, node) + node.nextSibling = None + return node + + def replaceChild(self, newChild, oldChild): + if newChild.nodeType == self.DOCUMENT_FRAGMENT_NODE: + refChild = oldChild.nextSibling + self.removeChild(oldChild) + return self.insertBefore(newChild, refChild) + if newChild.nodeType not in self._child_node_types: + raise xml.dom.HierarchyRequestErr( + "%s cannot be child of %s" % (repr(newChild), repr(self))) + if newChild is oldChild: + return + if newChild.parentNode is not None: + newChild.parentNode.removeChild(newChild) + try: + index = self.childNodes.index(oldChild) + except ValueError: + raise xml.dom.NotFoundErr() + self.childNodes[index] = newChild + newChild.parentNode = self + oldChild.parentNode = None + if (newChild.nodeType in _nodeTypes_with_children + or oldChild.nodeType in _nodeTypes_with_children): + _clear_id_cache(self) + newChild.nextSibling = oldChild.nextSibling + newChild.previousSibling = oldChild.previousSibling + oldChild.nextSibling = None + oldChild.previousSibling = None + if newChild.previousSibling: + newChild.previousSibling.nextSibling = newChild + if newChild.nextSibling: + newChild.nextSibling.previousSibling = newChild + return oldChild + + def removeChild(self, oldChild): + try: + self.childNodes.remove(oldChild) + except ValueError: + raise xml.dom.NotFoundErr() + if oldChild.nextSibling is not None: + oldChild.nextSibling.previousSibling = oldChild.previousSibling + if oldChild.previousSibling is not None: + oldChild.previousSibling.nextSibling = oldChild.nextSibling + oldChild.nextSibling = oldChild.previousSibling = None + if oldChild.nodeType in _nodeTypes_with_children: + _clear_id_cache(self) + + oldChild.parentNode = None + return oldChild + + def normalize(self): + L = [] + for child in self.childNodes: + if child.nodeType == Node.TEXT_NODE: + data = child.data + if data and L and L[-1].nodeType == child.nodeType: + # collapse text node + node = L[-1] + node.data = node.data + child.data + node.nextSibling = child.nextSibling + child.unlink() + elif data: + if L: + L[-1].nextSibling = child + child.previousSibling = L[-1] + else: + child.previousSibling = None + L.append(child) + else: + # empty text node; discard + child.unlink() + else: + if L: + L[-1].nextSibling = child + child.previousSibling = L[-1] + else: + child.previousSibling = None + L.append(child) + if child.nodeType == Node.ELEMENT_NODE: + child.normalize() + self.childNodes[:] = L + + def cloneNode(self, deep): + return _clone_node(self, deep, self.ownerDocument or self) + + def isSupported(self, feature, version): + return self.ownerDocument.implementation.hasFeature(feature, version) + + def _get_localName(self): + # Overridden in Element and Attr where localName can be Non-Null + return None + + # Node interfaces from Level 3 (WD 9 April 2002) + + def isSameNode(self, other): + return self is other + + def getInterface(self, feature): + if self.isSupported(feature, None): + return self + else: + return None + + # The "user data" functions use a dictionary that is only present + # if some user data has been set, so be careful not to assume it + # exists. + + def getUserData(self, key): + try: + return self._user_data[key][0] + except (AttributeError, KeyError): + return None + + def setUserData(self, key, data, handler): + old = None + try: + d = self._user_data + except AttributeError: + d = {} + self._user_data = d + if d.has_key(key): + old = d[key][0] + if data is None: + # ignore handlers passed for None + handler = None + if old is not None: + del d[key] + else: + d[key] = (data, handler) + return old + + def _call_user_data_handler(self, operation, src, dst): + if hasattr(self, "_user_data"): + for key, (data, handler) in self._user_data.items(): + if handler is not None: + handler.handle(operation, key, data, src, dst) + + # minidom-specific API: + + def unlink(self): + self.parentNode = self.ownerDocument = None + if self.childNodes: + for child in self.childNodes: + child.unlink() + self.childNodes = NodeList() + self.previousSibling = None + self.nextSibling = None + +defproperty(Node, "firstChild", doc="First child node, or None.") +defproperty(Node, "lastChild", doc="Last child node, or None.") +defproperty(Node, "localName", doc="Namespace-local name of this node.") + + +def _append_child(self, node): + # fast path with less checks; usable by DOM builders if careful + childNodes = self.childNodes + if childNodes: + last = childNodes[-1] + node.__dict__["previousSibling"] = last + last.__dict__["nextSibling"] = node + childNodes.append(node) + node.__dict__["parentNode"] = self + +def _in_document(node): + # return True iff node is part of a document tree + while node is not None: + if node.nodeType == Node.DOCUMENT_NODE: + return True + node = node.parentNode + return False + +def _write_data(writer, data): + "Writes datachars to writer." + data = data.replace("&", "&").replace("<", "<") + data = data.replace("\"", """).replace(">", ">") + writer.write(data) + +def _get_elements_by_tagName_helper(parent, name, rc): + for node in parent.childNodes: + if node.nodeType == Node.ELEMENT_NODE and \ + (name == "*" or node.tagName == name): + rc.append(node) + _get_elements_by_tagName_helper(node, name, rc) + return rc + +def _get_elements_by_tagName_ns_helper(parent, nsURI, localName, rc): + for node in parent.childNodes: + if node.nodeType == Node.ELEMENT_NODE: + if ((localName == "*" or node.localName == localName) and + (nsURI == "*" or node.namespaceURI == nsURI)): + rc.append(node) + _get_elements_by_tagName_ns_helper(node, nsURI, localName, rc) + return rc + +class DocumentFragment(Node): + nodeType = Node.DOCUMENT_FRAGMENT_NODE + nodeName = "#document-fragment" + nodeValue = None + attributes = None + parentNode = None + _child_node_types = (Node.ELEMENT_NODE, + Node.TEXT_NODE, + Node.CDATA_SECTION_NODE, + Node.ENTITY_REFERENCE_NODE, + Node.PROCESSING_INSTRUCTION_NODE, + Node.COMMENT_NODE, + Node.NOTATION_NODE) + + def __init__(self): + self.childNodes = NodeList() + + +class Attr(Node): + nodeType = Node.ATTRIBUTE_NODE + attributes = None + ownerElement = None + specified = False + _is_id = False + + _child_node_types = (Node.TEXT_NODE, Node.ENTITY_REFERENCE_NODE) + + def __init__(self, qName, namespaceURI=EMPTY_NAMESPACE, localName=None, + prefix=None): + # skip setattr for performance + d = self.__dict__ + d["nodeName"] = d["name"] = qName + d["namespaceURI"] = namespaceURI + d["prefix"] = prefix + d['childNodes'] = NodeList() + + # Add the single child node that represents the value of the attr + self.childNodes.append(Text()) + + # nodeValue and value are set elsewhere + + def _get_localName(self): + return self.nodeName.split(":", 1)[-1] + + def _get_name(self): + return self.name + + def _get_specified(self): + return self.specified + + def __setattr__(self, name, value): + d = self.__dict__ + if name in ("value", "nodeValue"): + d["value"] = d["nodeValue"] = value + d2 = self.childNodes[0].__dict__ + d2["data"] = d2["nodeValue"] = value + if self.ownerElement is not None: + _clear_id_cache(self.ownerElement) + elif name in ("name", "nodeName"): + d["name"] = d["nodeName"] = value + if self.ownerElement is not None: + _clear_id_cache(self.ownerElement) + else: + d[name] = value + + def _set_prefix(self, prefix): + nsuri = self.namespaceURI + if prefix == "xmlns": + if nsuri and nsuri != XMLNS_NAMESPACE: + raise xml.dom.NamespaceErr( + "illegal use of 'xmlns' prefix for the wrong namespace") + d = self.__dict__ + d['prefix'] = prefix + if prefix is None: + newName = self.localName + else: + newName = "%s:%s" % (prefix, self.localName) + if self.ownerElement: + _clear_id_cache(self.ownerElement) + d['nodeName'] = d['name'] = newName + + def _set_value(self, value): + d = self.__dict__ + d['value'] = d['nodeValue'] = value + if self.ownerElement: + _clear_id_cache(self.ownerElement) + self.childNodes[0].data = value + + def unlink(self): + # This implementation does not call the base implementation + # since most of that is not needed, and the expense of the + # method call is not warranted. We duplicate the removal of + # children, but that's all we needed from the base class. + elem = self.ownerElement + if elem is not None: + del elem._attrs[self.nodeName] + del elem._attrsNS[(self.namespaceURI, self.localName)] + if self._is_id: + self._is_id = False + elem._magic_id_nodes -= 1 + self.ownerDocument._magic_id_count -= 1 + for child in self.childNodes: + child.unlink() + del self.childNodes[:] + + def _get_isId(self): + if self._is_id: + return True + doc = self.ownerDocument + elem = self.ownerElement + if doc is None or elem is None: + return False + + info = doc._get_elem_info(elem) + if info is None: + return False + if self.namespaceURI: + return info.isIdNS(self.namespaceURI, self.localName) + else: + return info.isId(self.nodeName) + + def _get_schemaType(self): + doc = self.ownerDocument + elem = self.ownerElement + if doc is None or elem is None: + return _no_type + + info = doc._get_elem_info(elem) + if info is None: + return _no_type + if self.namespaceURI: + return info.getAttributeTypeNS(self.namespaceURI, self.localName) + else: + return info.getAttributeType(self.nodeName) + +defproperty(Attr, "isId", doc="True if this attribute is an ID.") +defproperty(Attr, "localName", doc="Namespace-local name of this attribute.") +defproperty(Attr, "schemaType", doc="Schema type for this attribute.") + + +class NamedNodeMap(object): + """The attribute list is a transient interface to the underlying + dictionaries. Mutations here will change the underlying element's + dictionary. + + Ordering is imposed artificially and does not reflect the order of + attributes as found in an input document. + """ + + __slots__ = ('_attrs', '_attrsNS', '_ownerElement') + + def __init__(self, attrs, attrsNS, ownerElement): + self._attrs = attrs + self._attrsNS = attrsNS + self._ownerElement = ownerElement + + def _get_length(self): + return len(self._attrs) + + def item(self, index): + try: + return self[self._attrs.keys()[index]] + except IndexError: + return None + + def items(self): + L = [] + for node in self._attrs.values(): + L.append((node.nodeName, node.value)) + return L + + def itemsNS(self): + L = [] + for node in self._attrs.values(): + L.append(((node.namespaceURI, node.localName), node.value)) + return L + + def has_key(self, key): + if isinstance(key, StringTypes): + return self._attrs.has_key(key) + else: + return self._attrsNS.has_key(key) + + def keys(self): + return self._attrs.keys() + + def keysNS(self): + return self._attrsNS.keys() + + def values(self): + return self._attrs.values() + + def get(self, name, value=None): + return self._attrs.get(name, value) + + __len__ = _get_length + + def __cmp__(self, other): + if self._attrs is getattr(other, "_attrs", None): + return 0 + else: + return cmp(id(self), id(other)) + + def __getitem__(self, attname_or_tuple): + if isinstance(attname_or_tuple, tuple): + return self._attrsNS[attname_or_tuple] + else: + return self._attrs[attname_or_tuple] + + # same as set + def __setitem__(self, attname, value): + if isinstance(value, StringTypes): + try: + node = self._attrs[attname] + except KeyError: + node = Attr(attname) + node.ownerDocument = self._ownerElement.ownerDocument + self.setNamedItem(node) + node.value = value + else: + if not isinstance(value, Attr): + raise TypeError, "value must be a string or Attr object" + node = value + self.setNamedItem(node) + + def getNamedItem(self, name): + try: + return self._attrs[name] + except KeyError: + return None + + def getNamedItemNS(self, namespaceURI, localName): + try: + return self._attrsNS[(namespaceURI, localName)] + except KeyError: + return None + + def removeNamedItem(self, name): + n = self.getNamedItem(name) + if n is not None: + _clear_id_cache(self._ownerElement) + del self._attrs[n.nodeName] + del self._attrsNS[(n.namespaceURI, n.localName)] + if n.__dict__.has_key('ownerElement'): + n.__dict__['ownerElement'] = None + return n + else: + raise xml.dom.NotFoundErr() + + def removeNamedItemNS(self, namespaceURI, localName): + n = self.getNamedItemNS(namespaceURI, localName) + if n is not None: + _clear_id_cache(self._ownerElement) + del self._attrsNS[(n.namespaceURI, n.localName)] + del self._attrs[n.nodeName] + if n.__dict__.has_key('ownerElement'): + n.__dict__['ownerElement'] = None + return n + else: + raise xml.dom.NotFoundErr() + + def setNamedItem(self, node): + if not isinstance(node, Attr): + raise xml.dom.HierarchyRequestErr( + "%s cannot be child of %s" % (repr(node), repr(self))) + old = self._attrs.get(node.name) + if old: + old.unlink() + self._attrs[node.name] = node + self._attrsNS[(node.namespaceURI, node.localName)] = node + node.ownerElement = self._ownerElement + _clear_id_cache(node.ownerElement) + return old + + def setNamedItemNS(self, node): + return self.setNamedItem(node) + + def __delitem__(self, attname_or_tuple): + node = self[attname_or_tuple] + _clear_id_cache(node.ownerElement) + node.unlink() + + def __getstate__(self): + return self._attrs, self._attrsNS, self._ownerElement + + def __setstate__(self, state): + self._attrs, self._attrsNS, self._ownerElement = state + +defproperty(NamedNodeMap, "length", + doc="Number of nodes in the NamedNodeMap.") + +AttributeList = NamedNodeMap + + +class TypeInfo(object): + __slots__ = 'namespace', 'name' + + def __init__(self, namespace, name): + self.namespace = namespace + self.name = name + + def __repr__(self): + if self.namespace: + return "" % (self.name, self.namespace) + else: + return "" % self.name + + def _get_name(self): + return self.name + + def _get_namespace(self): + return self.namespace + +_no_type = TypeInfo(None, None) + +class Element(Node): + nodeType = Node.ELEMENT_NODE + nodeValue = None + schemaType = _no_type + + _magic_id_nodes = 0 + + _child_node_types = (Node.ELEMENT_NODE, + Node.PROCESSING_INSTRUCTION_NODE, + Node.COMMENT_NODE, + Node.TEXT_NODE, + Node.CDATA_SECTION_NODE, + Node.ENTITY_REFERENCE_NODE) + + def __init__(self, tagName, namespaceURI=EMPTY_NAMESPACE, prefix=None, + localName=None): + self.tagName = self.nodeName = tagName + self.prefix = prefix + self.namespaceURI = namespaceURI + self.childNodes = NodeList() + + self._attrs = {} # attributes are double-indexed: + self._attrsNS = {} # tagName -> Attribute + # URI,localName -> Attribute + # in the future: consider lazy generation + # of attribute objects this is too tricky + # for now because of headaches with + # namespaces. + + def _get_localName(self): + return self.tagName.split(":", 1)[-1] + + def _get_tagName(self): + return self.tagName + + def unlink(self): + for attr in self._attrs.values(): + attr.unlink() + self._attrs = None + self._attrsNS = None + Node.unlink(self) + + def getAttribute(self, attname): + try: + return self._attrs[attname].value + except KeyError: + return "" + + def getAttributeNS(self, namespaceURI, localName): + try: + return self._attrsNS[(namespaceURI, localName)].value + except KeyError: + return "" + + def setAttribute(self, attname, value): + attr = self.getAttributeNode(attname) + if attr is None: + attr = Attr(attname) + # for performance + d = attr.__dict__ + d["value"] = d["nodeValue"] = value + d["ownerDocument"] = self.ownerDocument + self.setAttributeNode(attr) + elif value != attr.value: + d = attr.__dict__ + d["value"] = d["nodeValue"] = value + if attr.isId: + _clear_id_cache(self) + + def setAttributeNS(self, namespaceURI, qualifiedName, value): + prefix, localname = _nssplit(qualifiedName) + attr = self.getAttributeNodeNS(namespaceURI, localname) + if attr is None: + # for performance + attr = Attr(qualifiedName, namespaceURI, localname, prefix) + d = attr.__dict__ + d["prefix"] = prefix + d["nodeName"] = qualifiedName + d["value"] = d["nodeValue"] = value + d["ownerDocument"] = self.ownerDocument + self.setAttributeNode(attr) + else: + d = attr.__dict__ + if value != attr.value: + d["value"] = d["nodeValue"] = value + if attr.isId: + _clear_id_cache(self) + if attr.prefix != prefix: + d["prefix"] = prefix + d["nodeName"] = qualifiedName + + def getAttributeNode(self, attrname): + return self._attrs.get(attrname) + + def getAttributeNodeNS(self, namespaceURI, localName): + return self._attrsNS.get((namespaceURI, localName)) + + def setAttributeNode(self, attr): + if attr.ownerElement not in (None, self): + raise xml.dom.InuseAttributeErr("attribute node already owned") + old1 = self._attrs.get(attr.name, None) + if old1 is not None: + self.removeAttributeNode(old1) + old2 = self._attrsNS.get((attr.namespaceURI, attr.localName), None) + if old2 is not None and old2 is not old1: + self.removeAttributeNode(old2) + _set_attribute_node(self, attr) + + if old1 is not attr: + # It might have already been part of this node, in which case + # it doesn't represent a change, and should not be returned. + return old1 + if old2 is not attr: + return old2 + + setAttributeNodeNS = setAttributeNode + + def removeAttribute(self, name): + try: + attr = self._attrs[name] + except KeyError: + raise xml.dom.NotFoundErr() + self.removeAttributeNode(attr) + + def removeAttributeNS(self, namespaceURI, localName): + try: + attr = self._attrsNS[(namespaceURI, localName)] + except KeyError: + raise xml.dom.NotFoundErr() + self.removeAttributeNode(attr) + + def removeAttributeNode(self, node): + if node is None: + raise xml.dom.NotFoundErr() + try: + self._attrs[node.name] + except KeyError: + raise xml.dom.NotFoundErr() + _clear_id_cache(self) + node.unlink() + # Restore this since the node is still useful and otherwise + # unlinked + node.ownerDocument = self.ownerDocument + + removeAttributeNodeNS = removeAttributeNode + + def hasAttribute(self, name): + return self._attrs.has_key(name) + + def hasAttributeNS(self, namespaceURI, localName): + return self._attrsNS.has_key((namespaceURI, localName)) + + def getElementsByTagName(self, name): + return _get_elements_by_tagName_helper(self, name, NodeList()) + + def getElementsByTagNameNS(self, namespaceURI, localName): + return _get_elements_by_tagName_ns_helper( + self, namespaceURI, localName, NodeList()) + + def __repr__(self): + return "" % (self.tagName, id(self)) + + def writexml(self, writer, indent="", addindent="", newl=""): + # indent = current indentation + # addindent = indentation to add to higher levels + # newl = newline string + writer.write(indent+"<" + self.tagName) + + attrs = self._get_attributes() + a_names = attrs.keys() + a_names.sort() + + for a_name in a_names: + writer.write(" %s=\"" % a_name) + _write_data(writer, attrs[a_name].value) + writer.write("\"") + if self.childNodes: + writer.write(">%s"%(newl)) + for node in self.childNodes: + node.writexml(writer,indent+addindent,addindent,newl) + writer.write("%s%s" % (indent,self.tagName,newl)) + else: + writer.write("/>%s"%(newl)) + + def _get_attributes(self): + return NamedNodeMap(self._attrs, self._attrsNS, self) + + def hasAttributes(self): + if self._attrs: + return True + else: + return False + + # DOM Level 3 attributes, based on the 22 Oct 2002 draft + + def setIdAttribute(self, name): + idAttr = self.getAttributeNode(name) + self.setIdAttributeNode(idAttr) + + def setIdAttributeNS(self, namespaceURI, localName): + idAttr = self.getAttributeNodeNS(namespaceURI, localName) + self.setIdAttributeNode(idAttr) + + def setIdAttributeNode(self, idAttr): + if idAttr is None or not self.isSameNode(idAttr.ownerElement): + raise xml.dom.NotFoundErr() + if _get_containing_entref(self) is not None: + raise xml.dom.NoModificationAllowedErr() + if not idAttr._is_id: + idAttr.__dict__['_is_id'] = True + self._magic_id_nodes += 1 + self.ownerDocument._magic_id_count += 1 + _clear_id_cache(self) + +defproperty(Element, "attributes", + doc="NamedNodeMap of attributes on the element.") +defproperty(Element, "localName", + doc="Namespace-local name of this element.") + + +def _set_attribute_node(element, attr): + _clear_id_cache(element) + element._attrs[attr.name] = attr + element._attrsNS[(attr.namespaceURI, attr.localName)] = attr + + # This creates a circular reference, but Element.unlink() + # breaks the cycle since the references to the attribute + # dictionaries are tossed. + attr.__dict__['ownerElement'] = element + + +class Childless: + """Mixin that makes childless-ness easy to implement and avoids + the complexity of the Node methods that deal with children. + """ + + attributes = None + childNodes = EmptyNodeList() + firstChild = None + lastChild = None + + def _get_firstChild(self): + return None + + def _get_lastChild(self): + return None + + def appendChild(self, node): + raise xml.dom.HierarchyRequestErr( + self.nodeName + " nodes cannot have children") + + def hasChildNodes(self): + return False + + def insertBefore(self, newChild, refChild): + raise xml.dom.HierarchyRequestErr( + self.nodeName + " nodes do not have children") + + def removeChild(self, oldChild): + raise xml.dom.NotFoundErr( + self.nodeName + " nodes do not have children") + + def replaceChild(self, newChild, oldChild): + raise xml.dom.HierarchyRequestErr( + self.nodeName + " nodes do not have children") + + +class ProcessingInstruction(Childless, Node): + nodeType = Node.PROCESSING_INSTRUCTION_NODE + + def __init__(self, target, data): + self.target = self.nodeName = target + self.data = self.nodeValue = data + + def _get_data(self): + return self.data + def _set_data(self, value): + d = self.__dict__ + d['data'] = d['nodeValue'] = value + + def _get_target(self): + return self.target + def _set_target(self, value): + d = self.__dict__ + d['target'] = d['nodeName'] = value + + def __setattr__(self, name, value): + if name == "data" or name == "nodeValue": + self.__dict__['data'] = self.__dict__['nodeValue'] = value + elif name == "target" or name == "nodeName": + self.__dict__['target'] = self.__dict__['nodeName'] = value + else: + self.__dict__[name] = value + + def writexml(self, writer, indent="", addindent="", newl=""): + writer.write("%s%s" % (indent,self.target, self.data, newl)) + + +class CharacterData(Childless, Node): + def _get_length(self): + return len(self.data) + __len__ = _get_length + + def _get_data(self): + return self.__dict__['data'] + def _set_data(self, data): + d = self.__dict__ + d['data'] = d['nodeValue'] = data + + _get_nodeValue = _get_data + _set_nodeValue = _set_data + + def __setattr__(self, name, value): + if name == "data" or name == "nodeValue": + self.__dict__['data'] = self.__dict__['nodeValue'] = value + else: + self.__dict__[name] = value + + def __repr__(self): + data = self.data + if len(data) > 10: + dotdotdot = "..." + else: + dotdotdot = "" + return "" % ( + self.__class__.__name__, data[0:10], dotdotdot) + + def substringData(self, offset, count): + if offset < 0: + raise xml.dom.IndexSizeErr("offset cannot be negative") + if offset >= len(self.data): + raise xml.dom.IndexSizeErr("offset cannot be beyond end of data") + if count < 0: + raise xml.dom.IndexSizeErr("count cannot be negative") + return self.data[offset:offset+count] + + def appendData(self, arg): + self.data = self.data + arg + + def insertData(self, offset, arg): + if offset < 0: + raise xml.dom.IndexSizeErr("offset cannot be negative") + if offset >= len(self.data): + raise xml.dom.IndexSizeErr("offset cannot be beyond end of data") + if arg: + self.data = "%s%s%s" % ( + self.data[:offset], arg, self.data[offset:]) + + def deleteData(self, offset, count): + if offset < 0: + raise xml.dom.IndexSizeErr("offset cannot be negative") + if offset >= len(self.data): + raise xml.dom.IndexSizeErr("offset cannot be beyond end of data") + if count < 0: + raise xml.dom.IndexSizeErr("count cannot be negative") + if count: + self.data = self.data[:offset] + self.data[offset+count:] + + def replaceData(self, offset, count, arg): + if offset < 0: + raise xml.dom.IndexSizeErr("offset cannot be negative") + if offset >= len(self.data): + raise xml.dom.IndexSizeErr("offset cannot be beyond end of data") + if count < 0: + raise xml.dom.IndexSizeErr("count cannot be negative") + if count: + self.data = "%s%s%s" % ( + self.data[:offset], arg, self.data[offset+count:]) + +defproperty(CharacterData, "length", doc="Length of the string data.") + + +class Text(CharacterData): + # Make sure we don't add an instance __dict__ if we don't already + # have one, at least when that's possible: + # XXX this does not work, CharacterData is an old-style class + # __slots__ = () + + nodeType = Node.TEXT_NODE + nodeName = "#text" + attributes = None + + def splitText(self, offset): + if offset < 0 or offset > len(self.data): + raise xml.dom.IndexSizeErr("illegal offset value") + newText = self.__class__() + newText.data = self.data[offset:] + newText.ownerDocument = self.ownerDocument + next = self.nextSibling + if self.parentNode and self in self.parentNode.childNodes: + if next is None: + self.parentNode.appendChild(newText) + else: + self.parentNode.insertBefore(newText, next) + self.data = self.data[:offset] + return newText + + def writexml(self, writer, indent="", addindent="", newl=""): + _write_data(writer, "%s%s%s"%(indent, self.data, newl)) + + # DOM Level 3 (WD 9 April 2002) + + def _get_wholeText(self): + L = [self.data] + n = self.previousSibling + while n is not None: + if n.nodeType in (Node.TEXT_NODE, Node.CDATA_SECTION_NODE): + L.insert(0, n.data) + n = n.previousSibling + else: + break + n = self.nextSibling + while n is not None: + if n.nodeType in (Node.TEXT_NODE, Node.CDATA_SECTION_NODE): + L.append(n.data) + n = n.nextSibling + else: + break + return ''.join(L) + + def replaceWholeText(self, content): + # XXX This needs to be seriously changed if minidom ever + # supports EntityReference nodes. + parent = self.parentNode + n = self.previousSibling + while n is not None: + if n.nodeType in (Node.TEXT_NODE, Node.CDATA_SECTION_NODE): + next = n.previousSibling + parent.removeChild(n) + n = next + else: + break + n = self.nextSibling + if not content: + parent.removeChild(self) + while n is not None: + if n.nodeType in (Node.TEXT_NODE, Node.CDATA_SECTION_NODE): + next = n.nextSibling + parent.removeChild(n) + n = next + else: + break + if content: + d = self.__dict__ + d['data'] = content + d['nodeValue'] = content + return self + else: + return None + + def _get_isWhitespaceInElementContent(self): + if self.data.strip(): + return False + elem = _get_containing_element(self) + if elem is None: + return False + info = self.ownerDocument._get_elem_info(elem) + if info is None: + return False + else: + return info.isElementContent() + +defproperty(Text, "isWhitespaceInElementContent", + doc="True iff this text node contains only whitespace" + " and is in element content.") +defproperty(Text, "wholeText", + doc="The text of all logically-adjacent text nodes.") + + +def _get_containing_element(node): + c = node.parentNode + while c is not None: + if c.nodeType == Node.ELEMENT_NODE: + return c + c = c.parentNode + return None + +def _get_containing_entref(node): + c = node.parentNode + while c is not None: + if c.nodeType == Node.ENTITY_REFERENCE_NODE: + return c + c = c.parentNode + return None + + +class Comment(Childless, CharacterData): + nodeType = Node.COMMENT_NODE + nodeName = "#comment" + + def __init__(self, data): + self.data = self.nodeValue = data + + def writexml(self, writer, indent="", addindent="", newl=""): + writer.write("%s%s" % (indent, self.data, newl)) + + +class CDATASection(Text): + # Make sure we don't add an instance __dict__ if we don't already + # have one, at least when that's possible: + # XXX this does not work, Text is an old-style class + # __slots__ = () + + nodeType = Node.CDATA_SECTION_NODE + nodeName = "#cdata-section" + + def writexml(self, writer, indent="", addindent="", newl=""): + if self.data.find("]]>") >= 0: + raise ValueError("']]>' not allowed in a CDATA section") + writer.write("" % self.data) + + +class ReadOnlySequentialNamedNodeMap(object): + __slots__ = '_seq', + + def __init__(self, seq=()): + # seq should be a list or tuple + self._seq = seq + + def __len__(self): + return len(self._seq) + + def _get_length(self): + return len(self._seq) + + def getNamedItem(self, name): + for n in self._seq: + if n.nodeName == name: + return n + + def getNamedItemNS(self, namespaceURI, localName): + for n in self._seq: + if n.namespaceURI == namespaceURI and n.localName == localName: + return n + + def __getitem__(self, name_or_tuple): + if isinstance(name_or_tuple, tuple): + node = self.getNamedItemNS(*name_or_tuple) + else: + node = self.getNamedItem(name_or_tuple) + if node is None: + raise KeyError, name_or_tuple + return node + + def item(self, index): + if index < 0: + return None + try: + return self._seq[index] + except IndexError: + return None + + def removeNamedItem(self, name): + raise xml.dom.NoModificationAllowedErr( + "NamedNodeMap instance is read-only") + + def removeNamedItemNS(self, namespaceURI, localName): + raise xml.dom.NoModificationAllowedErr( + "NamedNodeMap instance is read-only") + + def setNamedItem(self, node): + raise xml.dom.NoModificationAllowedErr( + "NamedNodeMap instance is read-only") + + def setNamedItemNS(self, node): + raise xml.dom.NoModificationAllowedErr( + "NamedNodeMap instance is read-only") + + def __getstate__(self): + return [self._seq] + + def __setstate__(self, state): + self._seq = state[0] + +defproperty(ReadOnlySequentialNamedNodeMap, "length", + doc="Number of entries in the NamedNodeMap.") + + +class Identified: + """Mix-in class that supports the publicId and systemId attributes.""" + + # XXX this does not work, this is an old-style class + # __slots__ = 'publicId', 'systemId' + + def _identified_mixin_init(self, publicId, systemId): + self.publicId = publicId + self.systemId = systemId + + def _get_publicId(self): + return self.publicId + + def _get_systemId(self): + return self.systemId + +class DocumentType(Identified, Childless, Node): + nodeType = Node.DOCUMENT_TYPE_NODE + nodeValue = None + name = None + publicId = None + systemId = None + internalSubset = None + + def __init__(self, qualifiedName): + self.entities = ReadOnlySequentialNamedNodeMap() + self.notations = ReadOnlySequentialNamedNodeMap() + if qualifiedName: + prefix, localname = _nssplit(qualifiedName) + self.name = localname + self.nodeName = self.name + + def _get_internalSubset(self): + return self.internalSubset + + def cloneNode(self, deep): + if self.ownerDocument is None: + # it's ok + clone = DocumentType(None) + clone.name = self.name + clone.nodeName = self.name + operation = xml.dom.UserDataHandler.NODE_CLONED + if deep: + clone.entities._seq = [] + clone.notations._seq = [] + for n in self.notations._seq: + notation = Notation(n.nodeName, n.publicId, n.systemId) + clone.notations._seq.append(notation) + n._call_user_data_handler(operation, n, notation) + for e in self.entities._seq: + entity = Entity(e.nodeName, e.publicId, e.systemId, + e.notationName) + entity.actualEncoding = e.actualEncoding + entity.encoding = e.encoding + entity.version = e.version + clone.entities._seq.append(entity) + e._call_user_data_handler(operation, n, entity) + self._call_user_data_handler(operation, self, clone) + return clone + else: + return None + + def writexml(self, writer, indent="", addindent="", newl=""): + writer.write(""+newl) + +class Entity(Identified, Node): + attributes = None + nodeType = Node.ENTITY_NODE + nodeValue = None + + actualEncoding = None + encoding = None + version = None + + def __init__(self, name, publicId, systemId, notation): + self.nodeName = name + self.notationName = notation + self.childNodes = NodeList() + self._identified_mixin_init(publicId, systemId) + + def _get_actualEncoding(self): + return self.actualEncoding + + def _get_encoding(self): + return self.encoding + + def _get_version(self): + return self.version + + def appendChild(self, newChild): + raise xml.dom.HierarchyRequestErr( + "cannot append children to an entity node") + + def insertBefore(self, newChild, refChild): + raise xml.dom.HierarchyRequestErr( + "cannot insert children below an entity node") + + def removeChild(self, oldChild): + raise xml.dom.HierarchyRequestErr( + "cannot remove children from an entity node") + + def replaceChild(self, newChild, oldChild): + raise xml.dom.HierarchyRequestErr( + "cannot replace children of an entity node") + +class Notation(Identified, Childless, Node): + nodeType = Node.NOTATION_NODE + nodeValue = None + + def __init__(self, name, publicId, systemId): + self.nodeName = name + self._identified_mixin_init(publicId, systemId) + + +class DOMImplementation(DOMImplementationLS): + _features = [("core", "1.0"), + ("core", "2.0"), + ("core", "3.0"), + ("core", None), + ("xml", "1.0"), + ("xml", "2.0"), + ("xml", "3.0"), + ("xml", None), + ("ls-load", "3.0"), + ("ls-load", None), + ] + + def hasFeature(self, feature, version): + if version == "": + version = None + return (feature.lower(), version) in self._features + + def createDocument(self, namespaceURI, qualifiedName, doctype): + if doctype and doctype.parentNode is not None: + raise xml.dom.WrongDocumentErr( + "doctype object owned by another DOM tree") + doc = self._create_document() + + add_root_element = not (namespaceURI is None + and qualifiedName is None + and doctype is None) + + if not qualifiedName and add_root_element: + # The spec is unclear what to raise here; SyntaxErr + # would be the other obvious candidate. Since Xerces raises + # InvalidCharacterErr, and since SyntaxErr is not listed + # for createDocument, that seems to be the better choice. + # XXX: need to check for illegal characters here and in + # createElement. + + # DOM Level III clears this up when talking about the return value + # of this function. If namespaceURI, qName and DocType are + # Null the document is returned without a document element + # Otherwise if doctype or namespaceURI are not None + # Then we go back to the above problem + raise xml.dom.InvalidCharacterErr("Element with no name") + + if add_root_element: + prefix, localname = _nssplit(qualifiedName) + if prefix == "xml" \ + and namespaceURI != "http://www.w3.org/XML/1998/namespace": + raise xml.dom.NamespaceErr("illegal use of 'xml' prefix") + if prefix and not namespaceURI: + raise xml.dom.NamespaceErr( + "illegal use of prefix without namespaces") + element = doc.createElementNS(namespaceURI, qualifiedName) + if doctype: + doc.appendChild(doctype) + doc.appendChild(element) + + if doctype: + doctype.parentNode = doctype.ownerDocument = doc + + doc.doctype = doctype + doc.implementation = self + return doc + + def createDocumentType(self, qualifiedName, publicId, systemId): + doctype = DocumentType(qualifiedName) + doctype.publicId = publicId + doctype.systemId = systemId + return doctype + + # DOM Level 3 (WD 9 April 2002) + + def getInterface(self, feature): + if self.hasFeature(feature, None): + return self + else: + return None + + # internal + def _create_document(self): + return Document() + +class ElementInfo(object): + """Object that represents content-model information for an element. + + This implementation is not expected to be used in practice; DOM + builders should provide implementations which do the right thing + using information available to it. + + """ + + __slots__ = 'tagName', + + def __init__(self, name): + self.tagName = name + + def getAttributeType(self, aname): + return _no_type + + def getAttributeTypeNS(self, namespaceURI, localName): + return _no_type + + def isElementContent(self): + return False + + def isEmpty(self): + """Returns true iff this element is declared to have an EMPTY + content model.""" + return False + + def isId(self, aname): + """Returns true iff the named attribte is a DTD-style ID.""" + return False + + def isIdNS(self, namespaceURI, localName): + """Returns true iff the identified attribute is a DTD-style ID.""" + return False + + def __getstate__(self): + return self.tagName + + def __setstate__(self, state): + self.tagName = state + +def _clear_id_cache(node): + if node.nodeType == Node.DOCUMENT_NODE: + node._id_cache.clear() + node._id_search_stack = None + elif _in_document(node): + node.ownerDocument._id_cache.clear() + node.ownerDocument._id_search_stack= None + +class Document(Node, DocumentLS): + _child_node_types = (Node.ELEMENT_NODE, Node.PROCESSING_INSTRUCTION_NODE, + Node.COMMENT_NODE, Node.DOCUMENT_TYPE_NODE) + + nodeType = Node.DOCUMENT_NODE + nodeName = "#document" + nodeValue = None + attributes = None + doctype = None + parentNode = None + previousSibling = nextSibling = None + + implementation = DOMImplementation() + + # Document attributes from Level 3 (WD 9 April 2002) + + actualEncoding = None + encoding = None + standalone = None + version = None + strictErrorChecking = False + errorHandler = None + documentURI = None + + _magic_id_count = 0 + + def __init__(self): + self.childNodes = NodeList() + # mapping of (namespaceURI, localName) -> ElementInfo + # and tagName -> ElementInfo + self._elem_info = {} + self._id_cache = {} + self._id_search_stack = None + + def _get_elem_info(self, element): + if element.namespaceURI: + key = element.namespaceURI, element.localName + else: + key = element.tagName + return self._elem_info.get(key) + + def _get_actualEncoding(self): + return self.actualEncoding + + def _get_doctype(self): + return self.doctype + + def _get_documentURI(self): + return self.documentURI + + def _get_encoding(self): + return self.encoding + + def _get_errorHandler(self): + return self.errorHandler + + def _get_standalone(self): + return self.standalone + + def _get_strictErrorChecking(self): + return self.strictErrorChecking + + def _get_version(self): + return self.version + + def appendChild(self, node): + if node.nodeType not in self._child_node_types: + raise xml.dom.HierarchyRequestErr( + "%s cannot be child of %s" % (repr(node), repr(self))) + if node.parentNode is not None: + # This needs to be done before the next test since this + # may *be* the document element, in which case it should + # end up re-ordered to the end. + node.parentNode.removeChild(node) + + if node.nodeType == Node.ELEMENT_NODE \ + and self._get_documentElement(): + raise xml.dom.HierarchyRequestErr( + "two document elements disallowed") + return Node.appendChild(self, node) + + def removeChild(self, oldChild): + try: + self.childNodes.remove(oldChild) + except ValueError: + raise xml.dom.NotFoundErr() + oldChild.nextSibling = oldChild.previousSibling = None + oldChild.parentNode = None + if self.documentElement is oldChild: + self.documentElement = None + + return oldChild + + def _get_documentElement(self): + for node in self.childNodes: + if node.nodeType == Node.ELEMENT_NODE: + return node + + def unlink(self): + if self.doctype is not None: + self.doctype.unlink() + self.doctype = None + Node.unlink(self) + + def cloneNode(self, deep): + if not deep: + return None + clone = self.implementation.createDocument(None, None, None) + clone.encoding = self.encoding + clone.standalone = self.standalone + clone.version = self.version + for n in self.childNodes: + childclone = _clone_node(n, deep, clone) + assert childclone.ownerDocument.isSameNode(clone) + clone.childNodes.append(childclone) + if childclone.nodeType == Node.DOCUMENT_NODE: + assert clone.documentElement is None + elif childclone.nodeType == Node.DOCUMENT_TYPE_NODE: + assert clone.doctype is None + clone.doctype = childclone + childclone.parentNode = clone + self._call_user_data_handler(xml.dom.UserDataHandler.NODE_CLONED, + self, clone) + return clone + + def createDocumentFragment(self): + d = DocumentFragment() + d.ownerDocument = self + return d + + def createElement(self, tagName): + e = Element(tagName) + e.ownerDocument = self + return e + + def createTextNode(self, data): + if not isinstance(data, StringTypes): + raise TypeError, "node contents must be a string" + t = Text() + t.data = data + t.ownerDocument = self + return t + + def createCDATASection(self, data): + if not isinstance(data, StringTypes): + raise TypeError, "node contents must be a string" + c = CDATASection() + c.data = data + c.ownerDocument = self + return c + + def createComment(self, data): + c = Comment(data) + c.ownerDocument = self + return c + + def createProcessingInstruction(self, target, data): + p = ProcessingInstruction(target, data) + p.ownerDocument = self + return p + + def createAttribute(self, qName): + a = Attr(qName) + a.ownerDocument = self + a.value = "" + return a + + def createElementNS(self, namespaceURI, qualifiedName): + prefix, localName = _nssplit(qualifiedName) + e = Element(qualifiedName, namespaceURI, prefix) + e.ownerDocument = self + return e + + def createAttributeNS(self, namespaceURI, qualifiedName): + prefix, localName = _nssplit(qualifiedName) + a = Attr(qualifiedName, namespaceURI, localName, prefix) + a.ownerDocument = self + a.value = "" + return a + + # A couple of implementation-specific helpers to create node types + # not supported by the W3C DOM specs: + + def _create_entity(self, name, publicId, systemId, notationName): + e = Entity(name, publicId, systemId, notationName) + e.ownerDocument = self + return e + + def _create_notation(self, name, publicId, systemId): + n = Notation(name, publicId, systemId) + n.ownerDocument = self + return n + + def getElementById(self, id): + if self._id_cache.has_key(id): + return self._id_cache[id] + if not (self._elem_info or self._magic_id_count): + return None + + stack = self._id_search_stack + if stack is None: + # we never searched before, or the cache has been cleared + stack = [self.documentElement] + self._id_search_stack = stack + elif not stack: + # Previous search was completed and cache is still valid; + # no matching node. + return None + + result = None + while stack: + node = stack.pop() + # add child elements to stack for continued searching + stack.extend([child for child in node.childNodes + if child.nodeType in _nodeTypes_with_children]) + # check this node + info = self._get_elem_info(node) + if info: + # We have to process all ID attributes before + # returning in order to get all the attributes set to + # be IDs using Element.setIdAttribute*(). + for attr in node.attributes.values(): + if attr.namespaceURI: + if info.isIdNS(attr.namespaceURI, attr.localName): + self._id_cache[attr.value] = node + if attr.value == id: + result = node + elif not node._magic_id_nodes: + break + elif info.isId(attr.name): + self._id_cache[attr.value] = node + if attr.value == id: + result = node + elif not node._magic_id_nodes: + break + elif attr._is_id: + self._id_cache[attr.value] = node + if attr.value == id: + result = node + elif node._magic_id_nodes == 1: + break + elif node._magic_id_nodes: + for attr in node.attributes.values(): + if attr._is_id: + self._id_cache[attr.value] = node + if attr.value == id: + result = node + if result is not None: + break + return result + + def getElementsByTagName(self, name): + return _get_elements_by_tagName_helper(self, name, NodeList()) + + def getElementsByTagNameNS(self, namespaceURI, localName): + return _get_elements_by_tagName_ns_helper( + self, namespaceURI, localName, NodeList()) + + def isSupported(self, feature, version): + return self.implementation.hasFeature(feature, version) + + def importNode(self, node, deep): + if node.nodeType == Node.DOCUMENT_NODE: + raise xml.dom.NotSupportedErr("cannot import document nodes") + elif node.nodeType == Node.DOCUMENT_TYPE_NODE: + raise xml.dom.NotSupportedErr("cannot import document type nodes") + return _clone_node(node, deep, self) + + def writexml(self, writer, indent="", addindent="", newl="", + encoding = None): + if encoding is None: + writer.write(''+newl) + else: + writer.write('%s' % (encoding, newl)) + for node in self.childNodes: + node.writexml(writer, indent, addindent, newl) + + # DOM Level 3 (WD 9 April 2002) + + def renameNode(self, n, namespaceURI, name): + if n.ownerDocument is not self: + raise xml.dom.WrongDocumentErr( + "cannot rename nodes from other documents;\n" + "expected %s,\nfound %s" % (self, n.ownerDocument)) + if n.nodeType not in (Node.ELEMENT_NODE, Node.ATTRIBUTE_NODE): + raise xml.dom.NotSupportedErr( + "renameNode() only applies to element and attribute nodes") + if namespaceURI != EMPTY_NAMESPACE: + if ':' in name: + prefix, localName = name.split(':', 1) + if ( prefix == "xmlns" + and namespaceURI != xml.dom.XMLNS_NAMESPACE): + raise xml.dom.NamespaceErr( + "illegal use of 'xmlns' prefix") + else: + if ( name == "xmlns" + and namespaceURI != xml.dom.XMLNS_NAMESPACE + and n.nodeType == Node.ATTRIBUTE_NODE): + raise xml.dom.NamespaceErr( + "illegal use of the 'xmlns' attribute") + prefix = None + localName = name + else: + prefix = None + localName = None + if n.nodeType == Node.ATTRIBUTE_NODE: + element = n.ownerElement + if element is not None: + is_id = n._is_id + element.removeAttributeNode(n) + else: + element = None + # avoid __setattr__ + d = n.__dict__ + d['prefix'] = prefix + d['localName'] = localName + d['namespaceURI'] = namespaceURI + d['nodeName'] = name + if n.nodeType == Node.ELEMENT_NODE: + d['tagName'] = name + else: + # attribute node + d['name'] = name + if element is not None: + element.setAttributeNode(n) + if is_id: + element.setIdAttributeNode(n) + # It's not clear from a semantic perspective whether we should + # call the user data handlers for the NODE_RENAMED event since + # we're re-using the existing node. The draft spec has been + # interpreted as meaning "no, don't call the handler unless a + # new node is created." + return n + +defproperty(Document, "documentElement", + doc="Top-level element of this document.") + + +def _clone_node(node, deep, newOwnerDocument): + """ + Clone a node and give it the new owner document. + Called by Node.cloneNode and Document.importNode + """ + if node.ownerDocument.isSameNode(newOwnerDocument): + operation = xml.dom.UserDataHandler.NODE_CLONED + else: + operation = xml.dom.UserDataHandler.NODE_IMPORTED + if node.nodeType == Node.ELEMENT_NODE: + clone = newOwnerDocument.createElementNS(node.namespaceURI, + node.nodeName) + for attr in node.attributes.values(): + clone.setAttributeNS(attr.namespaceURI, attr.nodeName, attr.value) + a = clone.getAttributeNodeNS(attr.namespaceURI, attr.localName) + a.specified = attr.specified + + if deep: + for child in node.childNodes: + c = _clone_node(child, deep, newOwnerDocument) + clone.appendChild(c) + + elif node.nodeType == Node.DOCUMENT_FRAGMENT_NODE: + clone = newOwnerDocument.createDocumentFragment() + if deep: + for child in node.childNodes: + c = _clone_node(child, deep, newOwnerDocument) + clone.appendChild(c) + + elif node.nodeType == Node.TEXT_NODE: + clone = newOwnerDocument.createTextNode(node.data) + elif node.nodeType == Node.CDATA_SECTION_NODE: + clone = newOwnerDocument.createCDATASection(node.data) + elif node.nodeType == Node.PROCESSING_INSTRUCTION_NODE: + clone = newOwnerDocument.createProcessingInstruction(node.target, + node.data) + elif node.nodeType == Node.COMMENT_NODE: + clone = newOwnerDocument.createComment(node.data) + elif node.nodeType == Node.ATTRIBUTE_NODE: + clone = newOwnerDocument.createAttributeNS(node.namespaceURI, + node.nodeName) + clone.specified = True + clone.value = node.value + elif node.nodeType == Node.DOCUMENT_TYPE_NODE: + assert node.ownerDocument is not newOwnerDocument + operation = xml.dom.UserDataHandler.NODE_IMPORTED + clone = newOwnerDocument.implementation.createDocumentType( + node.name, node.publicId, node.systemId) + clone.ownerDocument = newOwnerDocument + if deep: + clone.entities._seq = [] + clone.notations._seq = [] + for n in node.notations._seq: + notation = Notation(n.nodeName, n.publicId, n.systemId) + notation.ownerDocument = newOwnerDocument + clone.notations._seq.append(notation) + if hasattr(n, '_call_user_data_handler'): + n._call_user_data_handler(operation, n, notation) + for e in node.entities._seq: + entity = Entity(e.nodeName, e.publicId, e.systemId, + e.notationName) + entity.actualEncoding = e.actualEncoding + entity.encoding = e.encoding + entity.version = e.version + entity.ownerDocument = newOwnerDocument + clone.entities._seq.append(entity) + if hasattr(e, '_call_user_data_handler'): + e._call_user_data_handler(operation, n, entity) + else: + # Note the cloning of Document and DocumentType nodes is + # implemenetation specific. minidom handles those cases + # directly in the cloneNode() methods. + raise xml.dom.NotSupportedErr("Cannot clone node %s" % repr(node)) + + # Check for _call_user_data_handler() since this could conceivably + # used with other DOM implementations (one of the FourThought + # DOMs, perhaps?). + if hasattr(node, '_call_user_data_handler'): + node._call_user_data_handler(operation, node, clone) + return clone + + +def _nssplit(qualifiedName): + fields = qualifiedName.split(':', 1) + if len(fields) == 2: + return fields + else: + return (None, fields[0]) + + +def _get_StringIO(): + # we can't use cStringIO since it doesn't support Unicode strings + from StringIO import StringIO + return StringIO() + +def _do_pulldom_parse(func, args, kwargs): + events = func(*args, **kwargs) + toktype, rootNode = events.getEvent() + events.expandNode(rootNode) + events.clear() + return rootNode + +def parse(file, parser=None, bufsize=None): + """Parse a file into a DOM by filename or file object.""" + if parser is None and not bufsize: + from xml.dom import expatbuilder + return expatbuilder.parse(file) + else: + from xml.dom import pulldom + return _do_pulldom_parse(pulldom.parse, (file,), + {'parser': parser, 'bufsize': bufsize}) + +def parseString(string, parser=None): + """Parse a file into a DOM from a string.""" + if parser is None: + from xml.dom import expatbuilder + return expatbuilder.parseString(string) + else: + from xml.dom import pulldom + return _do_pulldom_parse(pulldom.parseString, (string,), + {'parser': parser}) + +def getDOMImplementation(features=None): + if features: + if isinstance(features, StringTypes): + features = domreg._parse_feature_string(features) + for f, v in features: + if not Document.implementation.hasFeature(f, v): + return None + return Document.implementation Added: pypy/dist/pypy/lib/xml/dom/pulldom.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/pulldom.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,351 @@ +import xml.sax +import xml.sax.handler +import types + +try: + _StringTypes = [types.StringType, types.UnicodeType] +except AttributeError: + _StringTypes = [types.StringType] + +START_ELEMENT = "START_ELEMENT" +END_ELEMENT = "END_ELEMENT" +COMMENT = "COMMENT" +START_DOCUMENT = "START_DOCUMENT" +END_DOCUMENT = "END_DOCUMENT" +PROCESSING_INSTRUCTION = "PROCESSING_INSTRUCTION" +IGNORABLE_WHITESPACE = "IGNORABLE_WHITESPACE" +CHARACTERS = "CHARACTERS" + +class PullDOM(xml.sax.ContentHandler): + _locator = None + document = None + + def __init__(self, documentFactory=None): + from xml.dom import XML_NAMESPACE + self.documentFactory = documentFactory + self.firstEvent = [None, None] + self.lastEvent = self.firstEvent + self.elementStack = [] + self.push = self.elementStack.append + try: + self.pop = self.elementStack.pop + except AttributeError: + # use class' pop instead + pass + self._ns_contexts = [{XML_NAMESPACE:'xml'}] # contains uri -> prefix dicts + self._current_context = self._ns_contexts[-1] + self.pending_events = [] + + def pop(self): + result = self.elementStack[-1] + del self.elementStack[-1] + return result + + def setDocumentLocator(self, locator): + self._locator = locator + + def startPrefixMapping(self, prefix, uri): + if not hasattr(self, '_xmlns_attrs'): + self._xmlns_attrs = [] + self._xmlns_attrs.append((prefix or 'xmlns', uri)) + self._ns_contexts.append(self._current_context.copy()) + self._current_context[uri] = prefix or None + + def endPrefixMapping(self, prefix): + self._current_context = self._ns_contexts.pop() + + def startElementNS(self, name, tagName , attrs): + # Retrieve xml namespace declaration attributes. + xmlns_uri = 'http://www.w3.org/2000/xmlns/' + xmlns_attrs = getattr(self, '_xmlns_attrs', None) + if xmlns_attrs is not None: + for aname, value in xmlns_attrs: + attrs._attrs[(xmlns_uri, aname)] = value + self._xmlns_attrs = [] + uri, localname = name + if uri: + # When using namespaces, the reader may or may not + # provide us with the original name. If not, create + # *a* valid tagName from the current context. + if tagName is None: + prefix = self._current_context[uri] + if prefix: + tagName = prefix + ":" + localname + else: + tagName = localname + if self.document: + node = self.document.createElementNS(uri, tagName) + else: + node = self.buildDocument(uri, tagName) + else: + # When the tagname is not prefixed, it just appears as + # localname + if self.document: + node = self.document.createElement(localname) + else: + node = self.buildDocument(None, localname) + + for aname,value in attrs.items(): + a_uri, a_localname = aname + if a_uri == xmlns_uri: + if a_localname == 'xmlns': + qname = a_localname + else: + qname = 'xmlns:' + a_localname + attr = self.document.createAttributeNS(a_uri, qname) + node.setAttributeNodeNS(attr) + elif a_uri: + prefix = self._current_context[a_uri] + if prefix: + qname = prefix + ":" + a_localname + else: + qname = a_localname + attr = self.document.createAttributeNS(a_uri, qname) + node.setAttributeNodeNS(attr) + else: + attr = self.document.createAttribute(a_localname) + node.setAttributeNode(attr) + attr.value = value + + self.lastEvent[1] = [(START_ELEMENT, node), None] + self.lastEvent = self.lastEvent[1] + self.push(node) + + def endElementNS(self, name, tagName): + self.lastEvent[1] = [(END_ELEMENT, self.pop()), None] + self.lastEvent = self.lastEvent[1] + + def startElement(self, name, attrs): + if self.document: + node = self.document.createElement(name) + else: + node = self.buildDocument(None, name) + + for aname,value in attrs.items(): + attr = self.document.createAttribute(aname) + attr.value = value + node.setAttributeNode(attr) + + self.lastEvent[1] = [(START_ELEMENT, node), None] + self.lastEvent = self.lastEvent[1] + self.push(node) + + def endElement(self, name): + self.lastEvent[1] = [(END_ELEMENT, self.pop()), None] + self.lastEvent = self.lastEvent[1] + + def comment(self, s): + if self.document: + node = self.document.createComment(s) + self.lastEvent[1] = [(COMMENT, node), None] + self.lastEvent = self.lastEvent[1] + else: + event = [(COMMENT, s), None] + self.pending_events.append(event) + + def processingInstruction(self, target, data): + if self.document: + node = self.document.createProcessingInstruction(target, data) + self.lastEvent[1] = [(PROCESSING_INSTRUCTION, node), None] + self.lastEvent = self.lastEvent[1] + else: + event = [(PROCESSING_INSTRUCTION, target, data), None] + self.pending_events.append(event) + + def ignorableWhitespace(self, chars): + node = self.document.createTextNode(chars) + self.lastEvent[1] = [(IGNORABLE_WHITESPACE, node), None] + self.lastEvent = self.lastEvent[1] + + def characters(self, chars): + node = self.document.createTextNode(chars) + self.lastEvent[1] = [(CHARACTERS, node), None] + self.lastEvent = self.lastEvent[1] + + def startDocument(self): + if self.documentFactory is None: + import xml.dom.minidom + self.documentFactory = xml.dom.minidom.Document.implementation + + def buildDocument(self, uri, tagname): + # Can't do that in startDocument, since we need the tagname + # XXX: obtain DocumentType + node = self.documentFactory.createDocument(uri, tagname, None) + self.document = node + self.lastEvent[1] = [(START_DOCUMENT, node), None] + self.lastEvent = self.lastEvent[1] + self.push(node) + # Put everything we have seen so far into the document + for e in self.pending_events: + if e[0][0] == PROCESSING_INSTRUCTION: + _,target,data = e[0] + n = self.document.createProcessingInstruction(target, data) + e[0] = (PROCESSING_INSTRUCTION, n) + elif e[0][0] == COMMENT: + n = self.document.createComment(e[0][1]) + e[0] = (COMMENT, n) + else: + raise AssertionError("Unknown pending event ",e[0][0]) + self.lastEvent[1] = e + self.lastEvent = e + self.pending_events = None + return node.firstChild + + def endDocument(self): + self.lastEvent[1] = [(END_DOCUMENT, self.document), None] + self.pop() + + def clear(self): + "clear(): Explicitly release parsing structures" + self.document = None + +class ErrorHandler: + def warning(self, exception): + print exception + def error(self, exception): + raise exception + def fatalError(self, exception): + raise exception + +class DOMEventStream: + def __init__(self, stream, parser, bufsize): + self.stream = stream + self.parser = parser + self.bufsize = bufsize + if not hasattr(self.parser, 'feed'): + self.getEvent = self._slurp + self.reset() + + def reset(self): + self.pulldom = PullDOM() + # This content handler relies on namespace support + self.parser.setFeature(xml.sax.handler.feature_namespaces, 1) + self.parser.setContentHandler(self.pulldom) + + def __getitem__(self, pos): + rc = self.getEvent() + if rc: + return rc + raise IndexError + + def next(self): + rc = self.getEvent() + if rc: + return rc + raise StopIteration + + def __iter__(self): + return self + + def expandNode(self, node): + event = self.getEvent() + parents = [node] + while event: + token, cur_node = event + if cur_node is node: + return + if token != END_ELEMENT: + parents[-1].appendChild(cur_node) + if token == START_ELEMENT: + parents.append(cur_node) + elif token == END_ELEMENT: + del parents[-1] + event = self.getEvent() + + def getEvent(self): + # use IncrementalParser interface, so we get the desired + # pull effect + if not self.pulldom.firstEvent[1]: + self.pulldom.lastEvent = self.pulldom.firstEvent + while not self.pulldom.firstEvent[1]: + buf = self.stream.read(self.bufsize) + if not buf: + self.parser.close() + return None + self.parser.feed(buf) + rc = self.pulldom.firstEvent[1][0] + self.pulldom.firstEvent[1] = self.pulldom.firstEvent[1][1] + return rc + + def _slurp(self): + """ Fallback replacement for getEvent() using the + standard SAX2 interface, which means we slurp the + SAX events into memory (no performance gain, but + we are compatible to all SAX parsers). + """ + self.parser.parse(self.stream) + self.getEvent = self._emit + return self._emit() + + def _emit(self): + """ Fallback replacement for getEvent() that emits + the events that _slurp() read previously. + """ + rc = self.pulldom.firstEvent[1][0] + self.pulldom.firstEvent[1] = self.pulldom.firstEvent[1][1] + return rc + + def clear(self): + """clear(): Explicitly release parsing objects""" + self.pulldom.clear() + del self.pulldom + self.parser = None + self.stream = None + +class SAX2DOM(PullDOM): + + def startElementNS(self, name, tagName , attrs): + PullDOM.startElementNS(self, name, tagName, attrs) + curNode = self.elementStack[-1] + parentNode = self.elementStack[-2] + parentNode.appendChild(curNode) + + def startElement(self, name, attrs): + PullDOM.startElement(self, name, attrs) + curNode = self.elementStack[-1] + parentNode = self.elementStack[-2] + parentNode.appendChild(curNode) + + def processingInstruction(self, target, data): + PullDOM.processingInstruction(self, target, data) + node = self.lastEvent[0][1] + parentNode = self.elementStack[-1] + parentNode.appendChild(node) + + def ignorableWhitespace(self, chars): + PullDOM.ignorableWhitespace(self, chars) + node = self.lastEvent[0][1] + parentNode = self.elementStack[-1] + parentNode.appendChild(node) + + def characters(self, chars): + PullDOM.characters(self, chars) + node = self.lastEvent[0][1] + parentNode = self.elementStack[-1] + parentNode.appendChild(node) + + +default_bufsize = (2 ** 14) - 20 + +def parse(stream_or_string, parser=None, bufsize=None): + if bufsize is None: + bufsize = default_bufsize + if type(stream_or_string) in _StringTypes: + stream = open(stream_or_string) + else: + stream = stream_or_string + if not parser: + parser = xml.sax.make_parser() + return DOMEventStream(stream, parser, bufsize) + +def parseString(string, parser=None): + try: + from cStringIO import StringIO + except ImportError: + from StringIO import StringIO + + bufsize = len(string) + buf = StringIO(string) + if not parser: + parser = xml.sax.make_parser() + return DOMEventStream(buf, parser, bufsize) Added: pypy/dist/pypy/lib/xml/dom/xmlbuilder.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/dom/xmlbuilder.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,386 @@ +"""Implementation of the DOM Level 3 'LS-Load' feature.""" + +import copy +import xml.dom + +from xml.dom.NodeFilter import NodeFilter + + +__all__ = ["DOMBuilder", "DOMEntityResolver", "DOMInputSource"] + + +class Options: + """Features object that has variables set for each DOMBuilder feature. + + The DOMBuilder class uses an instance of this class to pass settings to + the ExpatBuilder class. + """ + + # Note that the DOMBuilder class in LoadSave constrains which of these + # values can be set using the DOM Level 3 LoadSave feature. + + namespaces = 1 + namespace_declarations = True + validation = False + external_parameter_entities = True + external_general_entities = True + external_dtd_subset = True + validate_if_schema = False + validate = False + datatype_normalization = False + create_entity_ref_nodes = True + entities = True + whitespace_in_element_content = True + cdata_sections = True + comments = True + charset_overrides_xml_encoding = True + infoset = False + supported_mediatypes_only = False + + errorHandler = None + filter = None + + +class DOMBuilder: + entityResolver = None + errorHandler = None + filter = None + + ACTION_REPLACE = 1 + ACTION_APPEND_AS_CHILDREN = 2 + ACTION_INSERT_AFTER = 3 + ACTION_INSERT_BEFORE = 4 + + _legal_actions = (ACTION_REPLACE, ACTION_APPEND_AS_CHILDREN, + ACTION_INSERT_AFTER, ACTION_INSERT_BEFORE) + + def __init__(self): + self._options = Options() + + def _get_entityResolver(self): + return self.entityResolver + def _set_entityResolver(self, entityResolver): + self.entityResolver = entityResolver + + def _get_errorHandler(self): + return self.errorHandler + def _set_errorHandler(self, errorHandler): + self.errorHandler = errorHandler + + def _get_filter(self): + return self.filter + def _set_filter(self, filter): + self.filter = filter + + def setFeature(self, name, state): + if self.supportsFeature(name): + state = state and 1 or 0 + try: + settings = self._settings[(_name_xform(name), state)] + except KeyError: + raise xml.dom.NotSupportedErr( + "unsupported feature: %r" % (name,)) + else: + for name, value in settings: + setattr(self._options, name, value) + else: + raise xml.dom.NotFoundErr("unknown feature: " + repr(name)) + + def supportsFeature(self, name): + return hasattr(self._options, _name_xform(name)) + + def canSetFeature(self, name, state): + key = (_name_xform(name), state and 1 or 0) + return self._settings.has_key(key) + + # This dictionary maps from (feature,value) to a list of + # (option,value) pairs that should be set on the Options object. + # If a (feature,value) setting is not in this dictionary, it is + # not supported by the DOMBuilder. + # + _settings = { + ("namespace_declarations", 0): [ + ("namespace_declarations", 0)], + ("namespace_declarations", 1): [ + ("namespace_declarations", 1)], + ("validation", 0): [ + ("validation", 0)], + ("external_general_entities", 0): [ + ("external_general_entities", 0)], + ("external_general_entities", 1): [ + ("external_general_entities", 1)], + ("external_parameter_entities", 0): [ + ("external_parameter_entities", 0)], + ("external_parameter_entities", 1): [ + ("external_parameter_entities", 1)], + ("validate_if_schema", 0): [ + ("validate_if_schema", 0)], + ("create_entity_ref_nodes", 0): [ + ("create_entity_ref_nodes", 0)], + ("create_entity_ref_nodes", 1): [ + ("create_entity_ref_nodes", 1)], + ("entities", 0): [ + ("create_entity_ref_nodes", 0), + ("entities", 0)], + ("entities", 1): [ + ("entities", 1)], + ("whitespace_in_element_content", 0): [ + ("whitespace_in_element_content", 0)], + ("whitespace_in_element_content", 1): [ + ("whitespace_in_element_content", 1)], + ("cdata_sections", 0): [ + ("cdata_sections", 0)], + ("cdata_sections", 1): [ + ("cdata_sections", 1)], + ("comments", 0): [ + ("comments", 0)], + ("comments", 1): [ + ("comments", 1)], + ("charset_overrides_xml_encoding", 0): [ + ("charset_overrides_xml_encoding", 0)], + ("charset_overrides_xml_encoding", 1): [ + ("charset_overrides_xml_encoding", 1)], + ("infoset", 0): [], + ("infoset", 1): [ + ("namespace_declarations", 0), + ("validate_if_schema", 0), + ("create_entity_ref_nodes", 0), + ("entities", 0), + ("cdata_sections", 0), + ("datatype_normalization", 1), + ("whitespace_in_element_content", 1), + ("comments", 1), + ("charset_overrides_xml_encoding", 1)], + ("supported_mediatypes_only", 0): [ + ("supported_mediatypes_only", 0)], + ("namespaces", 0): [ + ("namespaces", 0)], + ("namespaces", 1): [ + ("namespaces", 1)], + } + + def getFeature(self, name): + xname = _name_xform(name) + try: + return getattr(self._options, xname) + except AttributeError: + if name == "infoset": + options = self._options + return (options.datatype_normalization + and options.whitespace_in_element_content + and options.comments + and options.charset_overrides_xml_encoding + and not (options.namespace_declarations + or options.validate_if_schema + or options.create_entity_ref_nodes + or options.entities + or options.cdata_sections)) + raise xml.dom.NotFoundErr("feature %s not known" % repr(name)) + + def parseURI(self, uri): + if self.entityResolver: + input = self.entityResolver.resolveEntity(None, uri) + else: + input = DOMEntityResolver().resolveEntity(None, uri) + return self.parse(input) + + def parse(self, input): + options = copy.copy(self._options) + options.filter = self.filter + options.errorHandler = self.errorHandler + fp = input.byteStream + if fp is None and options.systemId: + import urllib2 + fp = urllib2.urlopen(input.systemId) + return self._parse_bytestream(fp, options) + + def parseWithContext(self, input, cnode, action): + if action not in self._legal_actions: + raise ValueError("not a legal action") + raise NotImplementedError("Haven't written this yet...") + + def _parse_bytestream(self, stream, options): + import xml.dom.expatbuilder + builder = xml.dom.expatbuilder.makeBuilder(options) + return builder.parseFile(stream) + + +def _name_xform(name): + return name.lower().replace('-', '_') + + +class DOMEntityResolver(object): + __slots__ = '_opener', + + def resolveEntity(self, publicId, systemId): + assert systemId is not None + source = DOMInputSource() + source.publicId = publicId + source.systemId = systemId + source.byteStream = self._get_opener().open(systemId) + + # determine the encoding if the transport provided it + source.encoding = self._guess_media_encoding(source) + + # determine the base URI is we can + import posixpath, urlparse + parts = urlparse.urlparse(systemId) + scheme, netloc, path, params, query, fragment = parts + # XXX should we check the scheme here as well? + if path and not path.endswith("/"): + path = posixpath.dirname(path) + "/" + parts = scheme, netloc, path, params, query, fragment + source.baseURI = urlparse.urlunparse(parts) + + return source + + def _get_opener(self): + try: + return self._opener + except AttributeError: + self._opener = self._create_opener() + return self._opener + + def _create_opener(self): + import urllib2 + return urllib2.build_opener() + + def _guess_media_encoding(self, source): + info = source.byteStream.info() + if info.has_key("Content-Type"): + for param in info.getplist(): + if param.startswith("charset="): + return param.split("=", 1)[1].lower() + + +class DOMInputSource(object): + __slots__ = ('byteStream', 'characterStream', 'stringData', + 'encoding', 'publicId', 'systemId', 'baseURI') + + def __init__(self): + self.byteStream = None + self.characterStream = None + self.stringData = None + self.encoding = None + self.publicId = None + self.systemId = None + self.baseURI = None + + def _get_byteStream(self): + return self.byteStream + def _set_byteStream(self, byteStream): + self.byteStream = byteStream + + def _get_characterStream(self): + return self.characterStream + def _set_characterStream(self, characterStream): + self.characterStream = characterStream + + def _get_stringData(self): + return self.stringData + def _set_stringData(self, data): + self.stringData = data + + def _get_encoding(self): + return self.encoding + def _set_encoding(self, encoding): + self.encoding = encoding + + def _get_publicId(self): + return self.publicId + def _set_publicId(self, publicId): + self.publicId = publicId + + def _get_systemId(self): + return self.systemId + def _set_systemId(self, systemId): + self.systemId = systemId + + def _get_baseURI(self): + return self.baseURI + def _set_baseURI(self, uri): + self.baseURI = uri + + +class DOMBuilderFilter: + """Element filter which can be used to tailor construction of + a DOM instance. + """ + + # There's really no need for this class; concrete implementations + # should just implement the endElement() and startElement() + # methods as appropriate. Using this makes it easy to only + # implement one of them. + + FILTER_ACCEPT = 1 + FILTER_REJECT = 2 + FILTER_SKIP = 3 + FILTER_INTERRUPT = 4 + + whatToShow = NodeFilter.SHOW_ALL + + def _get_whatToShow(self): + return self.whatToShow + + def acceptNode(self, element): + return self.FILTER_ACCEPT + + def startContainer(self, element): + return self.FILTER_ACCEPT + +del NodeFilter + + +class DocumentLS: + """Mixin to create documents that conform to the load/save spec.""" + + async = False + + def _get_async(self): + return False + def _set_async(self, async): + if async: + raise xml.dom.NotSupportedErr( + "asynchronous document loading is not supported") + + def abort(self): + # What does it mean to "clear" a document? Does the + # documentElement disappear? + raise NotImplementedError( + "haven't figured out what this means yet") + + def load(self, uri): + raise NotImplementedError("haven't written this yet") + + def loadXML(self, source): + raise NotImplementedError("haven't written this yet") + + def saveXML(self, snode): + if snode is None: + snode = self + elif snode.ownerDocument is not self: + raise xml.dom.WrongDocumentErr() + return snode.toxml() + + +class DOMImplementationLS: + MODE_SYNCHRONOUS = 1 + MODE_ASYNCHRONOUS = 2 + + def createDOMBuilder(self, mode, schemaType): + if schemaType is not None: + raise xml.dom.NotSupportedErr( + "schemaType not yet supported") + if mode == self.MODE_SYNCHRONOUS: + return DOMBuilder() + if mode == self.MODE_ASYNCHRONOUS: + raise xml.dom.NotSupportedErr( + "asynchronous builders are not supported") + raise ValueError("unknown value for mode") + + def createDOMWriter(self): + raise NotImplementedError( + "the writer interface hasn't been written yet!") + + def createDOMInputSource(self): + return DOMInputSource() Added: pypy/dist/pypy/lib/xml/etree/ElementInclude.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/etree/ElementInclude.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,143 @@ +# +# ElementTree +# $Id: ElementInclude.py 1862 2004-06-18 07:31:02Z Fredrik $ +# +# limited xinclude support for element trees +# +# history: +# 2003-08-15 fl created +# 2003-11-14 fl fixed default loader +# +# Copyright (c) 2003-2004 by Fredrik Lundh. All rights reserved. +# +# fredrik at pythonware.com +# http://www.pythonware.com +# +# -------------------------------------------------------------------- +# The ElementTree toolkit is +# +# Copyright (c) 1999-2004 by Fredrik Lundh +# +# By obtaining, using, and/or copying this software and/or its +# associated documentation, you agree that you have read, understood, +# and will comply with the following terms and conditions: +# +# Permission to use, copy, modify, and distribute this software and +# its associated documentation for any purpose and without fee is +# hereby granted, provided that the above copyright notice appears in +# all copies, and that both that copyright notice and this permission +# notice appear in supporting documentation, and that the name of +# Secret Labs AB or the author not be used in advertising or publicity +# pertaining to distribution of the software without specific, written +# prior permission. +# +# SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD +# TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT- +# ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR +# BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +# OF THIS SOFTWARE. +# -------------------------------------------------------------------- + +# Licensed to PSF under a Contributor Agreement. +# See http://www.python.org/2.4/license for licensing details. + +## +# Limited XInclude support for the ElementTree package. +## + +import copy +import ElementTree + +XINCLUDE = "{http://www.w3.org/2001/XInclude}" + +XINCLUDE_INCLUDE = XINCLUDE + "include" +XINCLUDE_FALLBACK = XINCLUDE + "fallback" + +## +# Fatal include error. + +class FatalIncludeError(SyntaxError): + pass + +## +# Default loader. This loader reads an included resource from disk. +# +# @param href Resource reference. +# @param parse Parse mode. Either "xml" or "text". +# @param encoding Optional text encoding. +# @return The expanded resource. If the parse mode is "xml", this +# is an ElementTree instance. If the parse mode is "text", this +# is a Unicode string. If the loader fails, it can return None +# or raise an IOError exception. +# @throws IOError If the loader fails to load the resource. + +def default_loader(href, parse, encoding=None): + file = open(href) + if parse == "xml": + data = ElementTree.parse(file).getroot() + else: + data = file.read() + if encoding: + data = data.decode(encoding) + file.close() + return data + +## +# Expand XInclude directives. +# +# @param elem Root element. +# @param loader Optional resource loader. If omitted, it defaults +# to {@link default_loader}. If given, it should be a callable +# that implements the same interface as default_loader. +# @throws FatalIncludeError If the function fails to include a given +# resource, or if the tree contains malformed XInclude elements. +# @throws IOError If the function fails to load a given resource. + +def include(elem, loader=None): + if loader is None: + loader = default_loader + # look for xinclude elements + i = 0 + while i < len(elem): + e = elem[i] + if e.tag == XINCLUDE_INCLUDE: + # process xinclude directive + href = e.get("href") + parse = e.get("parse", "xml") + if parse == "xml": + node = loader(href, parse) + if node is None: + raise FatalIncludeError( + "cannot load %r as %r" % (href, parse) + ) + node = copy.copy(node) + if e.tail: + node.tail = (node.tail or "") + e.tail + elem[i] = node + elif parse == "text": + text = loader(href, parse, e.get("encoding")) + if text is None: + raise FatalIncludeError( + "cannot load %r as %r" % (href, parse) + ) + if i: + node = elem[i-1] + node.tail = (node.tail or "") + text + else: + elem.text = (elem.text or "") + text + (e.tail or "") + del elem[i] + continue + else: + raise FatalIncludeError( + "unknown parse type in xi:include tag (%r)" % parse + ) + elif e.tag == XINCLUDE_FALLBACK: + raise FatalIncludeError( + "xi:fallback tag must be child of xi:include (%r)" % e.tag + ) + else: + include(e, loader) + i = i + 1 Added: pypy/dist/pypy/lib/xml/etree/ElementPath.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/etree/ElementPath.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,198 @@ +# +# ElementTree +# $Id: ElementPath.py 1858 2004-06-17 21:31:41Z Fredrik $ +# +# limited xpath support for element trees +# +# history: +# 2003-05-23 fl created +# 2003-05-28 fl added support for // etc +# 2003-08-27 fl fixed parsing of periods in element names +# +# Copyright (c) 2003-2004 by Fredrik Lundh. All rights reserved. +# +# fredrik at pythonware.com +# http://www.pythonware.com +# +# -------------------------------------------------------------------- +# The ElementTree toolkit is +# +# Copyright (c) 1999-2004 by Fredrik Lundh +# +# By obtaining, using, and/or copying this software and/or its +# associated documentation, you agree that you have read, understood, +# and will comply with the following terms and conditions: +# +# Permission to use, copy, modify, and distribute this software and +# its associated documentation for any purpose and without fee is +# hereby granted, provided that the above copyright notice appears in +# all copies, and that both that copyright notice and this permission +# notice appear in supporting documentation, and that the name of +# Secret Labs AB or the author not be used in advertising or publicity +# pertaining to distribution of the software without specific, written +# prior permission. +# +# SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD +# TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT- +# ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR +# BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +# OF THIS SOFTWARE. +# -------------------------------------------------------------------- + +# Licensed to PSF under a Contributor Agreement. +# See http://www.python.org/2.4/license for licensing details. + +## +# Implementation module for XPath support. There's usually no reason +# to import this module directly; the ElementTree does this for +# you, if needed. +## + +import re + +xpath_tokenizer = re.compile( + "(::|\.\.|\(\)|[/.*:\[\]\(\)@=])|((?:\{[^}]+\})?[^/:\[\]\(\)@=\s]+)|\s+" + ).findall + +class xpath_descendant_or_self: + pass + +## +# Wrapper for a compiled XPath. + +class Path: + + ## + # Create an Path instance from an XPath expression. + + def __init__(self, path): + tokens = xpath_tokenizer(path) + # the current version supports 'path/path'-style expressions only + self.path = [] + self.tag = None + if tokens and tokens[0][0] == "/": + raise SyntaxError("cannot use absolute path on element") + while tokens: + op, tag = tokens.pop(0) + if tag or op == "*": + self.path.append(tag or op) + elif op == ".": + pass + elif op == "/": + self.path.append(xpath_descendant_or_self()) + continue + else: + raise SyntaxError("unsupported path syntax (%s)" % op) + if tokens: + op, tag = tokens.pop(0) + if op != "/": + raise SyntaxError( + "expected path separator (%s)" % (op or tag) + ) + if self.path and isinstance(self.path[-1], xpath_descendant_or_self): + raise SyntaxError("path cannot end with //") + if len(self.path) == 1 and isinstance(self.path[0], type("")): + self.tag = self.path[0] + + ## + # Find first matching object. + + def find(self, element): + tag = self.tag + if tag is None: + nodeset = self.findall(element) + if not nodeset: + return None + return nodeset[0] + for elem in element: + if elem.tag == tag: + return elem + return None + + ## + # Find text for first matching object. + + def findtext(self, element, default=None): + tag = self.tag + if tag is None: + nodeset = self.findall(element) + if not nodeset: + return default + return nodeset[0].text or "" + for elem in element: + if elem.tag == tag: + return elem.text or "" + return default + + ## + # Find all matching objects. + + def findall(self, element): + nodeset = [element] + index = 0 + while 1: + try: + path = self.path[index] + index = index + 1 + except IndexError: + return nodeset + set = [] + if isinstance(path, xpath_descendant_or_self): + try: + tag = self.path[index] + if not isinstance(tag, type("")): + tag = None + else: + index = index + 1 + except IndexError: + tag = None # invalid path + for node in nodeset: + new = list(node.getiterator(tag)) + if new and new[0] is node: + set.extend(new[1:]) + else: + set.extend(new) + else: + for node in nodeset: + for node in node: + if path == "*" or node.tag == path: + set.append(node) + if not set: + return [] + nodeset = set + +_cache = {} + +## +# (Internal) Compile path. + +def _compile(path): + p = _cache.get(path) + if p is not None: + return p + p = Path(path) + if len(_cache) >= 100: + _cache.clear() + _cache[path] = p + return p + +## +# Find first matching object. + +def find(element, path): + return _compile(path).find(element) + +## +# Find text for first matching object. + +def findtext(element, path, default=None): + return _compile(path).findtext(element, default) + +## +# Find all matching objects. + +def findall(element, path): + return _compile(path).findall(element) Added: pypy/dist/pypy/lib/xml/etree/ElementTree.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/etree/ElementTree.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,1260 @@ +# +# ElementTree +# $Id: ElementTree.py 2326 2005-03-17 07:45:21Z fredrik $ +# +# light-weight XML support for Python 1.5.2 and later. +# +# history: +# 2001-10-20 fl created (from various sources) +# 2001-11-01 fl return root from parse method +# 2002-02-16 fl sort attributes in lexical order +# 2002-04-06 fl TreeBuilder refactoring, added PythonDoc markup +# 2002-05-01 fl finished TreeBuilder refactoring +# 2002-07-14 fl added basic namespace support to ElementTree.write +# 2002-07-25 fl added QName attribute support +# 2002-10-20 fl fixed encoding in write +# 2002-11-24 fl changed default encoding to ascii; fixed attribute encoding +# 2002-11-27 fl accept file objects or file names for parse/write +# 2002-12-04 fl moved XMLTreeBuilder back to this module +# 2003-01-11 fl fixed entity encoding glitch for us-ascii +# 2003-02-13 fl added XML literal factory +# 2003-02-21 fl added ProcessingInstruction/PI factory +# 2003-05-11 fl added tostring/fromstring helpers +# 2003-05-26 fl added ElementPath support +# 2003-07-05 fl added makeelement factory method +# 2003-07-28 fl added more well-known namespace prefixes +# 2003-08-15 fl fixed typo in ElementTree.findtext (Thomas Dartsch) +# 2003-09-04 fl fall back on emulator if ElementPath is not installed +# 2003-10-31 fl markup updates +# 2003-11-15 fl fixed nested namespace bug +# 2004-03-28 fl added XMLID helper +# 2004-06-02 fl added default support to findtext +# 2004-06-08 fl fixed encoding of non-ascii element/attribute names +# 2004-08-23 fl take advantage of post-2.1 expat features +# 2005-02-01 fl added iterparse implementation +# 2005-03-02 fl fixed iterparse support for pre-2.2 versions +# +# Copyright (c) 1999-2005 by Fredrik Lundh. All rights reserved. +# +# fredrik at pythonware.com +# http://www.pythonware.com +# +# -------------------------------------------------------------------- +# The ElementTree toolkit is +# +# Copyright (c) 1999-2005 by Fredrik Lundh +# +# By obtaining, using, and/or copying this software and/or its +# associated documentation, you agree that you have read, understood, +# and will comply with the following terms and conditions: +# +# Permission to use, copy, modify, and distribute this software and +# its associated documentation for any purpose and without fee is +# hereby granted, provided that the above copyright notice appears in +# all copies, and that both that copyright notice and this permission +# notice appear in supporting documentation, and that the name of +# Secret Labs AB or the author not be used in advertising or publicity +# pertaining to distribution of the software without specific, written +# prior permission. +# +# SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD +# TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT- +# ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR +# BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +# OF THIS SOFTWARE. +# -------------------------------------------------------------------- + +# Licensed to PSF under a Contributor Agreement. +# See http://www.python.org/2.4/license for licensing details. + +__all__ = [ + # public symbols + "Comment", + "dump", + "Element", "ElementTree", + "fromstring", + "iselement", "iterparse", + "parse", + "PI", "ProcessingInstruction", + "QName", + "SubElement", + "tostring", + "TreeBuilder", + "VERSION", "XML", + "XMLParser", "XMLTreeBuilder", + ] + +## +# The Element type is a flexible container object, designed to +# store hierarchical data structures in memory. The type can be +# described as a cross between a list and a dictionary. +#

+# Each element has a number of properties associated with it: +#

    +#
  • a tag. This is a string identifying what kind of data +# this element represents (the element type, in other words).
    • +#
  • a number of attributes, stored in a Python dictionary.
    • +#
  • a text string.
    • +#
  • an optional tail string.
    • +#
  • a number of child elements, stored in a Python sequence
    • +#
+# +# To create an element instance, use the {@link #Element} or {@link +# #SubElement} factory functions. +#

+# The {@link #ElementTree} class can be used to wrap an element +# structure, and convert it from and to XML. +## + +import string, sys, re + +class _SimpleElementPath: + # emulate pre-1.2 find/findtext/findall behaviour + def find(self, element, tag): + for elem in element: + if elem.tag == tag: + return elem + return None + def findtext(self, element, tag, default=None): + for elem in element: + if elem.tag == tag: + return elem.text or "" + return default + def findall(self, element, tag): + if tag[:3] == ".//": + return element.getiterator(tag[3:]) + result = [] + for elem in element: + if elem.tag == tag: + result.append(elem) + return result + +try: + import ElementPath +except ImportError: + # FIXME: issue warning in this case? + ElementPath = _SimpleElementPath() + +# TODO: add support for custom namespace resolvers/default namespaces +# TODO: add improved support for incremental parsing + +VERSION = "1.2.6" + +## +# Internal element class. This class defines the Element interface, +# and provides a reference implementation of this interface. +#

+# You should not create instances of this class directly. Use the +# appropriate factory functions instead, such as {@link #Element} +# and {@link #SubElement}. +# +# @see Element +# @see SubElement +# @see Comment +# @see ProcessingInstruction + +class _ElementInterface: + # text...tail + + ## + # (Attribute) Element tag. + + tag = None + + ## + # (Attribute) Element attribute dictionary. Where possible, use + # {@link #_ElementInterface.get}, + # {@link #_ElementInterface.set}, + # {@link #_ElementInterface.keys}, and + # {@link #_ElementInterface.items} to access + # element attributes. + + attrib = None + + ## + # (Attribute) Text before first subelement. This is either a + # string or the value None, if there was no text. + + text = None + + ## + # (Attribute) Text after this element's end tag, but before the + # next sibling element's start tag. This is either a string or + # the value None, if there was no text. + + tail = None # text after end tag, if any + + def __init__(self, tag, attrib): + self.tag = tag + self.attrib = attrib + self._children = [] + + def __repr__(self): + return "" % (self.tag, id(self)) + + ## + # Creates a new element object of the same type as this element. + # + # @param tag Element tag. + # @param attrib Element attributes, given as a dictionary. + # @return A new element instance. + + def makeelement(self, tag, attrib): + return Element(tag, attrib) + + ## + # Returns the number of subelements. + # + # @return The number of subelements. + + def __len__(self): + return len(self._children) + + ## + # Returns the given subelement. + # + # @param index What subelement to return. + # @return The given subelement. + # @exception IndexError If the given element does not exist. + + def __getitem__(self, index): + return self._children[index] + + ## + # Replaces the given subelement. + # + # @param index What subelement to replace. + # @param element The new element value. + # @exception IndexError If the given element does not exist. + # @exception AssertionError If element is not a valid object. + + def __setitem__(self, index, element): + assert iselement(element) + self._children[index] = element + + ## + # Deletes the given subelement. + # + # @param index What subelement to delete. + # @exception IndexError If the given element does not exist. + + def __delitem__(self, index): + del self._children[index] + + ## + # Returns a list containing subelements in the given range. + # + # @param start The first subelement to return. + # @param stop The first subelement that shouldn't be returned. + # @return A sequence object containing subelements. + + def __getslice__(self, start, stop): + return self._children[start:stop] + + ## + # Replaces a number of subelements with elements from a sequence. + # + # @param start The first subelement to replace. + # @param stop The first subelement that shouldn't be replaced. + # @param elements A sequence object with zero or more elements. + # @exception AssertionError If a sequence member is not a valid object. + + def __setslice__(self, start, stop, elements): + for element in elements: + assert iselement(element) + self._children[start:stop] = list(elements) + + ## + # Deletes a number of subelements. + # + # @param start The first subelement to delete. + # @param stop The first subelement to leave in there. + + def __delslice__(self, start, stop): + del self._children[start:stop] + + ## + # Adds a subelement to the end of this element. + # + # @param element The element to add. + # @exception AssertionError If a sequence member is not a valid object. + + def append(self, element): + assert iselement(element) + self._children.append(element) + + ## + # Inserts a subelement at the given position in this element. + # + # @param index Where to insert the new subelement. + # @exception AssertionError If the element is not a valid object. + + def insert(self, index, element): + assert iselement(element) + self._children.insert(index, element) + + ## + # Removes a matching subelement. Unlike the find methods, + # this method compares elements based on identity, not on tag + # value or contents. + # + # @param element What element to remove. + # @exception ValueError If a matching element could not be found. + # @exception AssertionError If the element is not a valid object. + + def remove(self, element): + assert iselement(element) + self._children.remove(element) + + ## + # Returns all subelements. The elements are returned in document + # order. + # + # @return A list of subelements. + # @defreturn list of Element instances + + def getchildren(self): + return self._children + + ## + # Finds the first matching subelement, by tag name or path. + # + # @param path What element to look for. + # @return The first matching element, or None if no element was found. + # @defreturn Element or None + + def find(self, path): + return ElementPath.find(self, path) + + ## + # Finds text for the first matching subelement, by tag name or path. + # + # @param path What element to look for. + # @param default What to return if the element was not found. + # @return The text content of the first matching element, or the + # default value no element was found. Note that if the element + # has is found, but has no text content, this method returns an + # empty string. + # @defreturn string + + def findtext(self, path, default=None): + return ElementPath.findtext(self, path, default) + + ## + # Finds all matching subelements, by tag name or path. + # + # @param path What element to look for. + # @return A list or iterator containing all matching elements, + # in document order. + # @defreturn list of Element instances + + def findall(self, path): + return ElementPath.findall(self, path) + + ## + # Resets an element. This function removes all subelements, clears + # all attributes, and sets the text and tail attributes to None. + + def clear(self): + self.attrib.clear() + self._children = [] + self.text = self.tail = None + + ## + # Gets an element attribute. + # + # @param key What attribute to look for. + # @param default What to return if the attribute was not found. + # @return The attribute value, or the default value, if the + # attribute was not found. + # @defreturn string or None + + def get(self, key, default=None): + return self.attrib.get(key, default) + + ## + # Sets an element attribute. + # + # @param key What attribute to set. + # @param value The attribute value. + + def set(self, key, value): + self.attrib[key] = value + + ## + # Gets a list of attribute names. The names are returned in an + # arbitrary order (just like for an ordinary Python dictionary). + # + # @return A list of element attribute names. + # @defreturn list of strings + + def keys(self): + return self.attrib.keys() + + ## + # Gets element attributes, as a sequence. The attributes are + # returned in an arbitrary order. + # + # @return A list of (name, value) tuples for all attributes. + # @defreturn list of (string, string) tuples + + def items(self): + return self.attrib.items() + + ## + # Creates a tree iterator. The iterator loops over this element + # and all subelements, in document order, and returns all elements + # with a matching tag. + #

+ # If the tree structure is modified during iteration, the result + # is undefined. + # + # @param tag What tags to look for (default is to return all elements). + # @return A list or iterator containing all the matching elements. + # @defreturn list or iterator + + def getiterator(self, tag=None): + nodes = [] + if tag == "*": + tag = None + if tag is None or self.tag == tag: + nodes.append(self) + for node in self._children: + nodes.extend(node.getiterator(tag)) + return nodes + +# compatibility +_Element = _ElementInterface + +## +# Element factory. This function returns an object implementing the +# standard Element interface. The exact class or type of that object +# is implementation dependent, but it will always be compatible with +# the {@link #_ElementInterface} class in this module. +#

+# The element name, attribute names, and attribute values can be +# either 8-bit ASCII strings or Unicode strings. +# +# @param tag The element name. +# @param attrib An optional dictionary, containing element attributes. +# @param **extra Additional attributes, given as keyword arguments. +# @return An element instance. +# @defreturn Element + +def Element(tag, attrib={}, **extra): + attrib = attrib.copy() + attrib.update(extra) + return _ElementInterface(tag, attrib) + +## +# Subelement factory. This function creates an element instance, and +# appends it to an existing element. +#

+# The element name, attribute names, and attribute values can be +# either 8-bit ASCII strings or Unicode strings. +# +# @param parent The parent element. +# @param tag The subelement name. +# @param attrib An optional dictionary, containing element attributes. +# @param **extra Additional attributes, given as keyword arguments. +# @return An element instance. +# @defreturn Element + +def SubElement(parent, tag, attrib={}, **extra): + attrib = attrib.copy() + attrib.update(extra) + element = parent.makeelement(tag, attrib) + parent.append(element) + return element + +## +# Comment element factory. This factory function creates a special +# element that will be serialized as an XML comment. +#

+# The comment string can be either an 8-bit ASCII string or a Unicode +# string. +# +# @param text A string containing the comment string. +# @return An element instance, representing a comment. +# @defreturn Element + +def Comment(text=None): + element = Element(Comment) + element.text = text + return element + +## +# PI element factory. This factory function creates a special element +# that will be serialized as an XML processing instruction. +# +# @param target A string containing the PI target. +# @param text A string containing the PI contents, if any. +# @return An element instance, representing a PI. +# @defreturn Element + +def ProcessingInstruction(target, text=None): + element = Element(ProcessingInstruction) + element.text = target + if text: + element.text = element.text + " " + text + return element + +PI = ProcessingInstruction + +## +# QName wrapper. This can be used to wrap a QName attribute value, in +# order to get proper namespace handling on output. +# +# @param text A string containing the QName value, in the form {uri}local, +# or, if the tag argument is given, the URI part of a QName. +# @param tag Optional tag. If given, the first argument is interpreted as +# an URI, and this argument is interpreted as a local name. +# @return An opaque object, representing the QName. + +class QName: + def __init__(self, text_or_uri, tag=None): + if tag: + text_or_uri = "{%s}%s" % (text_or_uri, tag) + self.text = text_or_uri + def __str__(self): + return self.text + def __hash__(self): + return hash(self.text) + def __cmp__(self, other): + if isinstance(other, QName): + return cmp(self.text, other.text) + return cmp(self.text, other) + +## +# ElementTree wrapper class. This class represents an entire element +# hierarchy, and adds some extra support for serialization to and from +# standard XML. +# +# @param element Optional root element. +# @keyparam file Optional file handle or name. If given, the +# tree is initialized with the contents of this XML file. + +class ElementTree: + + def __init__(self, element=None, file=None): + assert element is None or iselement(element) + self._root = element # first node + if file: + self.parse(file) + + ## + # Gets the root element for this tree. + # + # @return An element instance. + # @defreturn Element + + def getroot(self): + return self._root + + ## + # Replaces the root element for this tree. This discards the + # current contents of the tree, and replaces it with the given + # element. Use with care. + # + # @param element An element instance. + + def _setroot(self, element): + assert iselement(element) + self._root = element + + ## + # Loads an external XML document into this element tree. + # + # @param source A file name or file object. + # @param parser An optional parser instance. If not given, the + # standard {@link XMLTreeBuilder} parser is used. + # @return The document root element. + # @defreturn Element + + def parse(self, source, parser=None): + if not hasattr(source, "read"): + source = open(source, "rb") + if not parser: + parser = XMLTreeBuilder() + while 1: + data = source.read(32768) + if not data: + break + parser.feed(data) + self._root = parser.close() + return self._root + + ## + # Creates a tree iterator for the root element. The iterator loops + # over all elements in this tree, in document order. + # + # @param tag What tags to look for (default is to return all elements) + # @return An iterator. + # @defreturn iterator + + def getiterator(self, tag=None): + assert self._root is not None + return self._root.getiterator(tag) + + ## + # Finds the first toplevel element with given tag. + # Same as getroot().find(path). + # + # @param path What element to look for. + # @return The first matching element, or None if no element was found. + # @defreturn Element or None + + def find(self, path): + assert self._root is not None + if path[:1] == "/": + path = "." + path + return self._root.find(path) + + ## + # Finds the element text for the first toplevel element with given + # tag. Same as getroot().findtext(path). + # + # @param path What toplevel element to look for. + # @param default What to return if the element was not found. + # @return The text content of the first matching element, or the + # default value no element was found. Note that if the element + # has is found, but has no text content, this method returns an + # empty string. + # @defreturn string + + def findtext(self, path, default=None): + assert self._root is not None + if path[:1] == "/": + path = "." + path + return self._root.findtext(path, default) + + ## + # Finds all toplevel elements with the given tag. + # Same as getroot().findall(path). + # + # @param path What element to look for. + # @return A list or iterator containing all matching elements, + # in document order. + # @defreturn list of Element instances + + def findall(self, path): + assert self._root is not None + if path[:1] == "/": + path = "." + path + return self._root.findall(path) + + ## + # Writes the element tree to a file, as XML. + # + # @param file A file name, or a file object opened for writing. + # @param encoding Optional output encoding (default is US-ASCII). + + def write(self, file, encoding="us-ascii"): + assert self._root is not None + if not hasattr(file, "write"): + file = open(file, "wb") + if not encoding: + encoding = "us-ascii" + elif encoding != "utf-8" and encoding != "us-ascii": + file.write("\n" % encoding) + self._write(file, self._root, encoding, {}) + + def _write(self, file, node, encoding, namespaces): + # write XML to file + tag = node.tag + if tag is Comment: + file.write("" % _escape_cdata(node.text, encoding)) + elif tag is ProcessingInstruction: + file.write("" % _escape_cdata(node.text, encoding)) + else: + items = node.items() + xmlns_items = [] # new namespaces in this scope + try: + if isinstance(tag, QName) or tag[:1] == "{": + tag, xmlns = fixtag(tag, namespaces) + if xmlns: xmlns_items.append(xmlns) + except TypeError: + _raise_serialization_error(tag) + file.write("<" + _encode(tag, encoding)) + if items or xmlns_items: + items.sort() # lexical order + for k, v in items: + try: + if isinstance(k, QName) or k[:1] == "{": + k, xmlns = fixtag(k, namespaces) + if xmlns: xmlns_items.append(xmlns) + except TypeError: + _raise_serialization_error(k) + try: + if isinstance(v, QName): + v, xmlns = fixtag(v, namespaces) + if xmlns: xmlns_items.append(xmlns) + except TypeError: + _raise_serialization_error(v) + file.write(" %s=\"%s\"" % (_encode(k, encoding), + _escape_attrib(v, encoding))) + for k, v in xmlns_items: + file.write(" %s=\"%s\"" % (_encode(k, encoding), + _escape_attrib(v, encoding))) + if node.text or len(node): + file.write(">") + if node.text: + file.write(_escape_cdata(node.text, encoding)) + for n in node: + self._write(file, n, encoding, namespaces) + file.write("") + else: + file.write(" />") + for k, v in xmlns_items: + del namespaces[v] + if node.tail: + file.write(_escape_cdata(node.tail, encoding)) + +# -------------------------------------------------------------------- +# helpers + +## +# Checks if an object appears to be a valid element object. +# +# @param An element instance. +# @return A true value if this is an element object. +# @defreturn flag + +def iselement(element): + # FIXME: not sure about this; might be a better idea to look + # for tag/attrib/text attributes + return isinstance(element, _ElementInterface) or hasattr(element, "tag") + +## +# Writes an element tree or element structure to sys.stdout. This +# function should be used for debugging only. +#

+# The exact output format is implementation dependent. In this +# version, it's written as an ordinary XML file. +# +# @param elem An element tree or an individual element. + +def dump(elem): + # debugging + if not isinstance(elem, ElementTree): + elem = ElementTree(elem) + elem.write(sys.stdout) + tail = elem.getroot().tail + if not tail or tail[-1] != "\n": + sys.stdout.write("\n") + +def _encode(s, encoding): + try: + return s.encode(encoding) + except AttributeError: + return s # 1.5.2: assume the string uses the right encoding + +if sys.version[:3] == "1.5": + _escape = re.compile(r"[&<>\"\x80-\xff]+") # 1.5.2 +else: + _escape = re.compile(eval(r'u"[&<>\"\u0080-\uffff]+"')) + +_escape_map = { + "&": "&", + "<": "<", + ">": ">", + '"': """, +} + +_namespace_map = { + # "well-known" namespace prefixes + "http://www.w3.org/XML/1998/namespace": "xml", + "http://www.w3.org/1999/xhtml": "html", + "http://www.w3.org/1999/02/22-rdf-syntax-ns#": "rdf", + "http://schemas.xmlsoap.org/wsdl/": "wsdl", +} + +def _raise_serialization_error(text): + raise TypeError( + "cannot serialize %r (type %s)" % (text, type(text).__name__) + ) + +def _encode_entity(text, pattern=_escape): + # map reserved and non-ascii characters to numerical entities + def escape_entities(m, map=_escape_map): + out = [] + append = out.append + for char in m.group(): + text = map.get(char) + if text is None: + text = "&#%d;" % ord(char) + append(text) + return string.join(out, "") + try: + return _encode(pattern.sub(escape_entities, text), "ascii") + except TypeError: + _raise_serialization_error(text) + +# +# the following functions assume an ascii-compatible encoding +# (or "utf-16") + +def _escape_cdata(text, encoding=None, replace=string.replace): + # escape character data + try: + if encoding: + try: + text = _encode(text, encoding) + except UnicodeError: + return _encode_entity(text) + text = replace(text, "&", "&") + text = replace(text, "<", "<") + text = replace(text, ">", ">") + return text + except (TypeError, AttributeError): + _raise_serialization_error(text) + +def _escape_attrib(text, encoding=None, replace=string.replace): + # escape attribute value + try: + if encoding: + try: + text = _encode(text, encoding) + except UnicodeError: + return _encode_entity(text) + text = replace(text, "&", "&") + text = replace(text, "'", "'") # FIXME: overkill + text = replace(text, "\"", """) + text = replace(text, "<", "<") + text = replace(text, ">", ">") + return text + except (TypeError, AttributeError): + _raise_serialization_error(text) + +def fixtag(tag, namespaces): + # given a decorated tag (of the form {uri}tag), return prefixed + # tag and namespace declaration, if any + if isinstance(tag, QName): + tag = tag.text + namespace_uri, tag = string.split(tag[1:], "}", 1) + prefix = namespaces.get(namespace_uri) + if prefix is None: + prefix = _namespace_map.get(namespace_uri) + if prefix is None: + prefix = "ns%d" % len(namespaces) + namespaces[namespace_uri] = prefix + if prefix == "xml": + xmlns = None + else: + xmlns = ("xmlns:%s" % prefix, namespace_uri) + else: + xmlns = None + return "%s:%s" % (prefix, tag), xmlns + +## +# Parses an XML document into an element tree. +# +# @param source A filename or file object containing XML data. +# @param parser An optional parser instance. If not given, the +# standard {@link XMLTreeBuilder} parser is used. +# @return An ElementTree instance + +def parse(source, parser=None): + tree = ElementTree() + tree.parse(source, parser) + return tree + +## +# Parses an XML document into an element tree incrementally, and reports +# what's going on to the user. +# +# @param source A filename or file object containing XML data. +# @param events A list of events to report back. If omitted, only "end" +# events are reported. +# @return A (event, elem) iterator. + +class iterparse: + + def __init__(self, source, events=None): + if not hasattr(source, "read"): + source = open(source, "rb") + self._file = source + self._events = [] + self._index = 0 + self.root = self._root = None + self._parser = XMLTreeBuilder() + # wire up the parser for event reporting + parser = self._parser._parser + append = self._events.append + if events is None: + events = ["end"] + for event in events: + if event == "start": + try: + parser.ordered_attributes = 1 + parser.specified_attributes = 1 + def handler(tag, attrib_in, event=event, append=append, + start=self._parser._start_list): + append((event, start(tag, attrib_in))) + parser.StartElementHandler = handler + except AttributeError: + def handler(tag, attrib_in, event=event, append=append, + start=self._parser._start): + append((event, start(tag, attrib_in))) + parser.StartElementHandler = handler + elif event == "end": + def handler(tag, event=event, append=append, + end=self._parser._end): + append((event, end(tag))) + parser.EndElementHandler = handler + elif event == "start-ns": + def handler(prefix, uri, event=event, append=append): + try: + uri = _encode(uri, "ascii") + except UnicodeError: + pass + append((event, (prefix or "", uri))) + parser.StartNamespaceDeclHandler = handler + elif event == "end-ns": + def handler(prefix, event=event, append=append): + append((event, None)) + parser.EndNamespaceDeclHandler = handler + + def next(self): + while 1: + try: + item = self._events[self._index] + except IndexError: + if self._parser is None: + self.root = self._root + try: + raise StopIteration + except NameError: + raise IndexError + # load event buffer + del self._events[:] + self._index = 0 + data = self._file.read(16384) + if data: + self._parser.feed(data) + else: + self._root = self._parser.close() + self._parser = None + else: + self._index = self._index + 1 + return item + + try: + iter + def __iter__(self): + return self + except NameError: + def __getitem__(self, index): + return self.next() + +## +# Parses an XML document from a string constant. This function can +# be used to embed "XML literals" in Python code. +# +# @param source A string containing XML data. +# @return An Element instance. +# @defreturn Element + +def XML(text): + parser = XMLTreeBuilder() + parser.feed(text) + return parser.close() + +## +# Parses an XML document from a string constant, and also returns +# a dictionary which maps from element id:s to elements. +# +# @param source A string containing XML data. +# @return A tuple containing an Element instance and a dictionary. +# @defreturn (Element, dictionary) + +def XMLID(text): + parser = XMLTreeBuilder() + parser.feed(text) + tree = parser.close() + ids = {} + for elem in tree.getiterator(): + id = elem.get("id") + if id: + ids[id] = elem + return tree, ids + +## +# Parses an XML document from a string constant. Same as {@link #XML}. +# +# @def fromstring(text) +# @param source A string containing XML data. +# @return An Element instance. +# @defreturn Element + +fromstring = XML + +## +# Generates a string representation of an XML element, including all +# subelements. +# +# @param element An Element instance. +# @return An encoded string containing the XML data. +# @defreturn string + +def tostring(element, encoding=None): + class dummy: + pass + data = [] + file = dummy() + file.write = data.append + ElementTree(element).write(file, encoding) + return string.join(data, "") + +## +# Generic element structure builder. This builder converts a sequence +# of {@link #TreeBuilder.start}, {@link #TreeBuilder.data}, and {@link +# #TreeBuilder.end} method calls to a well-formed element structure. +#

+# You can use this class to build an element structure using a custom XML +# parser, or a parser for some other XML-like format. +# +# @param element_factory Optional element factory. This factory +# is called to create new Element instances, as necessary. + +class TreeBuilder: + + def __init__(self, element_factory=None): + self._data = [] # data collector + self._elem = [] # element stack + self._last = None # last element + self._tail = None # true if we're after an end tag + if element_factory is None: + element_factory = _ElementInterface + self._factory = element_factory + + ## + # Flushes the parser buffers, and returns the toplevel documen + # element. + # + # @return An Element instance. + # @defreturn Element + + def close(self): + assert len(self._elem) == 0, "missing end tags" + assert self._last != None, "missing toplevel element" + return self._last + + def _flush(self): + if self._data: + if self._last is not None: + text = string.join(self._data, "") + if self._tail: + assert self._last.tail is None, "internal error (tail)" + self._last.tail = text + else: + assert self._last.text is None, "internal error (text)" + self._last.text = text + self._data = [] + + ## + # Adds text to the current element. + # + # @param data A string. This should be either an 8-bit string + # containing ASCII text, or a Unicode string. + + def data(self, data): + self._data.append(data) + + ## + # Opens a new element. + # + # @param tag The element name. + # @param attrib A dictionary containing element attributes. + # @return The opened element. + # @defreturn Element + + def start(self, tag, attrs): + self._flush() + self._last = elem = self._factory(tag, attrs) + if self._elem: + self._elem[-1].append(elem) + self._elem.append(elem) + self._tail = 0 + return elem + + ## + # Closes the current element. + # + # @param tag The element name. + # @return The closed element. + # @defreturn Element + + def end(self, tag): + self._flush() + self._last = self._elem.pop() + assert self._last.tag == tag,\ + "end tag mismatch (expected %s, got %s)" % ( + self._last.tag, tag) + self._tail = 1 + return self._last + +## +# Element structure builder for XML source data, based on the +# expat parser. +# +# @keyparam target Target object. If omitted, the builder uses an +# instance of the standard {@link #TreeBuilder} class. +# @keyparam html Predefine HTML entities. This flag is not supported +# by the current implementation. +# @see #ElementTree +# @see #TreeBuilder + +class XMLTreeBuilder: + + def __init__(self, html=0, target=None): + try: + from xml.parsers import expat + except ImportError: + raise ImportError( + "No module named expat; use SimpleXMLTreeBuilder instead" + ) + self._parser = parser = expat.ParserCreate(None, "}") + if target is None: + target = TreeBuilder() + self._target = target + self._names = {} # name memo cache + # callbacks + parser.DefaultHandlerExpand = self._default + parser.StartElementHandler = self._start + parser.EndElementHandler = self._end + parser.CharacterDataHandler = self._data + # let expat do the buffering, if supported + try: + self._parser.buffer_text = 1 + except AttributeError: + pass + # use new-style attribute handling, if supported + try: + self._parser.ordered_attributes = 1 + self._parser.specified_attributes = 1 + parser.StartElementHandler = self._start_list + except AttributeError: + pass + encoding = None + if not parser.returns_unicode: + encoding = "utf-8" + # target.xml(encoding, None) + self._doctype = None + self.entity = {} + + def _fixtext(self, text): + # convert text string to ascii, if possible + try: + return _encode(text, "ascii") + except UnicodeError: + return text + + def _fixname(self, key): + # expand qname, and convert name string to ascii, if possible + try: + name = self._names[key] + except KeyError: + name = key + if "}" in name: + name = "{" + name + self._names[key] = name = self._fixtext(name) + return name + + def _start(self, tag, attrib_in): + fixname = self._fixname + tag = fixname(tag) + attrib = {} + for key, value in attrib_in.items(): + attrib[fixname(key)] = self._fixtext(value) + return self._target.start(tag, attrib) + + def _start_list(self, tag, attrib_in): + fixname = self._fixname + tag = fixname(tag) + attrib = {} + if attrib_in: + for i in range(0, len(attrib_in), 2): + attrib[fixname(attrib_in[i])] = self._fixtext(attrib_in[i+1]) + return self._target.start(tag, attrib) + + def _data(self, text): + return self._target.data(self._fixtext(text)) + + def _end(self, tag): + return self._target.end(self._fixname(tag)) + + def _default(self, text): + prefix = text[:1] + if prefix == "&": + # deal with undefined entities + try: + self._target.data(self.entity[text[1:-1]]) + except KeyError: + from xml.parsers import expat + raise expat.error( + "undefined entity %s: line %d, column %d" % + (text, self._parser.ErrorLineNumber, + self._parser.ErrorColumnNumber) + ) + elif prefix == "<" and text[:9] == "": + self._doctype = None + return + text = string.strip(text) + if not text: + return + self._doctype.append(text) + n = len(self._doctype) + if n > 2: + type = self._doctype[1] + if type == "PUBLIC" and n == 4: + name, type, pubid, system = self._doctype + elif type == "SYSTEM" and n == 3: + name, type, system = self._doctype + pubid = None + else: + return + if pubid: + pubid = pubid[1:-1] + self.doctype(name, pubid, system[1:-1]) + self._doctype = None + + ## + # Handles a doctype declaration. + # + # @param name Doctype name. + # @param pubid Public identifier. + # @param system System identifier. + + def doctype(self, name, pubid, system): + pass + + ## + # Feeds data to the parser. + # + # @param data Encoded data. + + def feed(self, data): + self._parser.Parse(data, 0) + + ## + # Finishes feeding data to the parser. + # + # @return An element structure. + # @defreturn Element + + def close(self): + self._parser.Parse("", 1) # end of data + tree = self._target.close() + del self._target, self._parser # get rid of circular references + return tree + +# compatibility +XMLParser = XMLTreeBuilder Added: pypy/dist/pypy/lib/xml/etree/__init__.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/etree/__init__.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,33 @@ +# $Id: __init__.py 1821 2004-06-03 16:57:49Z fredrik $ +# elementtree package + +# -------------------------------------------------------------------- +# The ElementTree toolkit is +# +# Copyright (c) 1999-2004 by Fredrik Lundh +# +# By obtaining, using, and/or copying this software and/or its +# associated documentation, you agree that you have read, understood, +# and will comply with the following terms and conditions: +# +# Permission to use, copy, modify, and distribute this software and +# its associated documentation for any purpose and without fee is +# hereby granted, provided that the above copyright notice appears in +# all copies, and that both that copyright notice and this permission +# notice appear in supporting documentation, and that the name of +# Secret Labs AB or the author not be used in advertising or publicity +# pertaining to distribution of the software without specific, written +# prior permission. +# +# SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD +# TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT- +# ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR +# BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +# OF THIS SOFTWARE. +# -------------------------------------------------------------------- + +# Licensed to PSF under a Contributor Agreement. +# See http://www.python.org/2.4/license for licensing details. Added: pypy/dist/pypy/lib/xml/etree/cElementTree.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/etree/cElementTree.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,3 @@ +# Wrapper module for _elementtree + +from _elementtree import * Added: pypy/dist/pypy/lib/xml/parsers/__init__.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/parsers/__init__.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,8 @@ +"""Python interfaces to XML parsers. + +This package contains one module: + +expat -- Python wrapper for James Clark's Expat parser, with namespace + support. + +""" Added: pypy/dist/pypy/lib/xml/parsers/expat.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/parsers/expat.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,4 @@ +"""Interface to the Expat non-validating XML parser.""" +__version__ = '$Revision: 17640 $' + +from pyexpat import * Added: pypy/dist/pypy/lib/xml/sax/__init__.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/sax/__init__.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,108 @@ +"""Simple API for XML (SAX) implementation for Python. + +This module provides an implementation of the SAX 2 interface; +information about the Java version of the interface can be found at +http://www.megginson.com/SAX/. The Python version of the interface is +documented at <...>. + +This package contains the following modules: + +handler -- Base classes and constants which define the SAX 2 API for + the 'client-side' of SAX for Python. + +saxutils -- Implementation of the convenience classes commonly used to + work with SAX. + +xmlreader -- Base classes and constants which define the SAX 2 API for + the parsers used with SAX for Python. + +expatreader -- Driver that allows use of the Expat parser with SAX. +""" + +from xmlreader import InputSource +from handler import ContentHandler, ErrorHandler +from _exceptions import SAXException, SAXNotRecognizedException, \ + SAXParseException, SAXNotSupportedException, \ + SAXReaderNotAvailable + + +def parse(source, handler, errorHandler=ErrorHandler()): + parser = make_parser() + parser.setContentHandler(handler) + parser.setErrorHandler(errorHandler) + parser.parse(source) + +def parseString(string, handler, errorHandler=ErrorHandler()): + try: + from cStringIO import StringIO + except ImportError: + from StringIO import StringIO + + if errorHandler is None: + errorHandler = ErrorHandler() + parser = make_parser() + parser.setContentHandler(handler) + parser.setErrorHandler(errorHandler) + + inpsrc = InputSource() + inpsrc.setByteStream(StringIO(string)) + parser.parse(inpsrc) + +# this is the parser list used by the make_parser function if no +# alternatives are given as parameters to the function + +default_parser_list = ["xml.sax.expatreader"] + +# tell modulefinder that importing sax potentially imports expatreader +_false = 0 +if _false: + import xml.sax.expatreader + +import os, sys +if os.environ.has_key("PY_SAX_PARSER"): + default_parser_list = os.environ["PY_SAX_PARSER"].split(",") +del os + +_key = "python.xml.sax.parser" +if sys.platform[:4] == "java" and sys.registry.containsKey(_key): + default_parser_list = sys.registry.getProperty(_key).split(",") + + +def make_parser(parser_list = []): + """Creates and returns a SAX parser. + + Creates the first parser it is able to instantiate of the ones + given in the list created by doing parser_list + + default_parser_list. The lists must contain the names of Python + modules containing both a SAX parser and a create_parser function.""" + + for parser_name in parser_list + default_parser_list: + try: + return _create_parser(parser_name) + except ImportError,e: + import sys + if sys.modules.has_key(parser_name): + # The parser module was found, but importing it + # failed unexpectedly, pass this exception through + raise + except SAXReaderNotAvailable: + # The parser module detected that it won't work properly, + # so try the next one + pass + + raise SAXReaderNotAvailable("No parsers found", None) + +# --- Internal utility methods used by make_parser + +if sys.platform[ : 4] == "java": + def _create_parser(parser_name): + from org.python.core import imp + drv_module = imp.importName(parser_name, 0, globals()) + return drv_module.create_parser() + +else: + def _create_parser(parser_name): + drv_module = __import__(parser_name,{},{},['create_parser']) + return drv_module.create_parser() + +del sys Added: pypy/dist/pypy/lib/xml/sax/_exceptions.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/sax/_exceptions.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,131 @@ +"""Different kinds of SAX Exceptions""" +import sys +if sys.platform[:4] == "java": + from java.lang import Exception +del sys + +# ===== SAXEXCEPTION ===== + +class SAXException(Exception): + """Encapsulate an XML error or warning. This class can contain + basic error or warning information from either the XML parser or + the application: you can subclass it to provide additional + functionality, or to add localization. Note that although you will + receive a SAXException as the argument to the handlers in the + ErrorHandler interface, you are not actually required to throw + the exception; instead, you can simply read the information in + it.""" + + def __init__(self, msg, exception=None): + """Creates an exception. The message is required, but the exception + is optional.""" + self._msg = msg + self._exception = exception + Exception.__init__(self, msg) + + def getMessage(self): + "Return a message for this exception." + return self._msg + + def getException(self): + "Return the embedded exception, or None if there was none." + return self._exception + + def __str__(self): + "Create a string representation of the exception." + return self._msg + + def __getitem__(self, ix): + """Avoids weird error messages if someone does exception[ix] by + mistake, since Exception has __getitem__ defined.""" + raise AttributeError("__getitem__") + + +# ===== SAXPARSEEXCEPTION ===== + +class SAXParseException(SAXException): + """Encapsulate an XML parse error or warning. + + This exception will include information for locating the error in + the original XML document. Note that although the application will + receive a SAXParseException as the argument to the handlers in the + ErrorHandler interface, the application is not actually required + to throw the exception; instead, it can simply read the + information in it and take a different action. + + Since this exception is a subclass of SAXException, it inherits + the ability to wrap another exception.""" + + def __init__(self, msg, exception, locator): + "Creates the exception. The exception parameter is allowed to be None." + SAXException.__init__(self, msg, exception) + self._locator = locator + + # We need to cache this stuff at construction time. + # If this exception is thrown, the objects through which we must + # traverse to get this information may be deleted by the time + # it gets caught. + self._systemId = self._locator.getSystemId() + self._colnum = self._locator.getColumnNumber() + self._linenum = self._locator.getLineNumber() + + def getColumnNumber(self): + """The column number of the end of the text where the exception + occurred.""" + return self._colnum + + def getLineNumber(self): + "The line number of the end of the text where the exception occurred." + return self._linenum + + def getPublicId(self): + "Get the public identifier of the entity where the exception occurred." + return self._locator.getPublicId() + + def getSystemId(self): + "Get the system identifier of the entity where the exception occurred." + return self._systemId + + def __str__(self): + "Create a string representation of the exception." + sysid = self.getSystemId() + if sysid is None: + sysid = "" + linenum = self.getLineNumber() + if linenum is None: + linenum = "?" + colnum = self.getColumnNumber() + if colnum is None: + colnum = "?" + return "%s:%s:%s: %s" % (sysid, linenum, colnum, self._msg) + + +# ===== SAXNOTRECOGNIZEDEXCEPTION ===== + +class SAXNotRecognizedException(SAXException): + """Exception class for an unrecognized identifier. + + An XMLReader will raise this exception when it is confronted with an + unrecognized feature or property. SAX applications and extensions may + use this class for similar purposes.""" + + +# ===== SAXNOTSUPPORTEDEXCEPTION ===== + +class SAXNotSupportedException(SAXException): + """Exception class for an unsupported operation. + + An XMLReader will raise this exception when a service it cannot + perform is requested (specifically setting a state or value). SAX + applications and extensions may use this class for similar + purposes.""" + +# ===== SAXNOTSUPPORTEDEXCEPTION ===== + +class SAXReaderNotAvailable(SAXNotSupportedException): + """Exception class for a missing driver. + + An XMLReader module (driver) should raise this exception when it + is first imported, e.g. when a support module cannot be imported. + It also may be raised during parsing, e.g. if executing an external + program is not permitted.""" Added: pypy/dist/pypy/lib/xml/sax/expatreader.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/sax/expatreader.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,414 @@ +""" +SAX driver for the pyexpat C module. This driver works with +pyexpat.__version__ == '2.22'. +""" + +version = "0.20" + +from xml.sax._exceptions import * +from xml.sax.handler import feature_validation, feature_namespaces +from xml.sax.handler import feature_namespace_prefixes +from xml.sax.handler import feature_external_ges, feature_external_pes +from xml.sax.handler import feature_string_interning +from xml.sax.handler import property_xml_string, property_interning_dict + +# xml.parsers.expat does not raise ImportError in Jython +import sys +if sys.platform[:4] == "java": + raise SAXReaderNotAvailable("expat not available in Java", None) +del sys + +try: + from xml.parsers import expat +except ImportError: + raise SAXReaderNotAvailable("expat not supported", None) +else: + if not hasattr(expat, "ParserCreate"): + raise SAXReaderNotAvailable("expat not supported", None) +from xml.sax import xmlreader, saxutils, handler + +AttributesImpl = xmlreader.AttributesImpl +AttributesNSImpl = xmlreader.AttributesNSImpl + +# If we're using a sufficiently recent version of Python, we can use +# weak references to avoid cycles between the parser and content +# handler, otherwise we'll just have to pretend. +try: + import _weakref +except ImportError: + def _mkproxy(o): + return o +else: + import weakref + _mkproxy = weakref.proxy + del weakref, _weakref + +# --- ExpatLocator + +class ExpatLocator(xmlreader.Locator): + """Locator for use with the ExpatParser class. + + This uses a weak reference to the parser object to avoid creating + a circular reference between the parser and the content handler. + """ + def __init__(self, parser): + self._ref = _mkproxy(parser) + + def getColumnNumber(self): + parser = self._ref + if parser._parser is None: + return None + return parser._parser.ErrorColumnNumber + + def getLineNumber(self): + parser = self._ref + if parser._parser is None: + return 1 + return parser._parser.ErrorLineNumber + + def getPublicId(self): + parser = self._ref + if parser is None: + return None + return parser._source.getPublicId() + + def getSystemId(self): + parser = self._ref + if parser is None: + return None + return parser._source.getSystemId() + + +# --- ExpatParser + +class ExpatParser(xmlreader.IncrementalParser, xmlreader.Locator): + """SAX driver for the pyexpat C module.""" + + def __init__(self, namespaceHandling=0, bufsize=2**16-20): + xmlreader.IncrementalParser.__init__(self, bufsize) + self._source = xmlreader.InputSource() + self._parser = None + self._namespaces = namespaceHandling + self._lex_handler_prop = None + self._parsing = 0 + self._entity_stack = [] + self._external_ges = 1 + self._interning = None + + # XMLReader methods + + def parse(self, source): + "Parse an XML document from a URL or an InputSource." + source = saxutils.prepare_input_source(source) + + self._source = source + self.reset() + self._cont_handler.setDocumentLocator(ExpatLocator(self)) + xmlreader.IncrementalParser.parse(self, source) + + def prepareParser(self, source): + if source.getSystemId() != None: + self._parser.SetBase(source.getSystemId()) + + # Redefined setContentHandler to allow changing handlers during parsing + + def setContentHandler(self, handler): + xmlreader.IncrementalParser.setContentHandler(self, handler) + if self._parsing: + self._reset_cont_handler() + + def getFeature(self, name): + if name == feature_namespaces: + return self._namespaces + elif name == feature_string_interning: + return self._interning is not None + elif name in (feature_validation, feature_external_pes, + feature_namespace_prefixes): + return 0 + elif name == feature_external_ges: + return self._external_ges + raise SAXNotRecognizedException("Feature '%s' not recognized" % name) + + def setFeature(self, name, state): + if self._parsing: + raise SAXNotSupportedException("Cannot set features while parsing") + + if name == feature_namespaces: + self._namespaces = state + elif name == feature_external_ges: + self._external_ges = state + elif name == feature_string_interning: + if state: + if self._interning is None: + self._interning = {} + else: + self._interning = None + elif name == feature_validation: + if state: + raise SAXNotSupportedException( + "expat does not support validation") + elif name == feature_external_pes: + if state: + raise SAXNotSupportedException( + "expat does not read external parameter entities") + elif name == feature_namespace_prefixes: + if state: + raise SAXNotSupportedException( + "expat does not report namespace prefixes") + else: + raise SAXNotRecognizedException( + "Feature '%s' not recognized" % name) + + def getProperty(self, name): + if name == handler.property_lexical_handler: + return self._lex_handler_prop + elif name == property_interning_dict: + return self._interning + elif name == property_xml_string: + if self._parser: + if hasattr(self._parser, "GetInputContext"): + return self._parser.GetInputContext() + else: + raise SAXNotRecognizedException( + "This version of expat does not support getting" + " the XML string") + else: + raise SAXNotSupportedException( + "XML string cannot be returned when not parsing") + raise SAXNotRecognizedException("Property '%s' not recognized" % name) + + def setProperty(self, name, value): + if name == handler.property_lexical_handler: + self._lex_handler_prop = value + if self._parsing: + self._reset_lex_handler_prop() + elif name == property_interning_dict: + self._interning = value + elif name == property_xml_string: + raise SAXNotSupportedException("Property '%s' cannot be set" % + name) + else: + raise SAXNotRecognizedException("Property '%s' not recognized" % + name) + + # IncrementalParser methods + + def feed(self, data, isFinal = 0): + if not self._parsing: + self.reset() + self._parsing = 1 + self._cont_handler.startDocument() + + try: + # The isFinal parameter is internal to the expat reader. + # If it is set to true, expat will check validity of the entire + # document. When feeding chunks, they are not normally final - + # except when invoked from close. + self._parser.Parse(data, isFinal) + except expat.error, e: + exc = SAXParseException(expat.ErrorString(e.code), e, self) + # FIXME: when to invoke error()? + self._err_handler.fatalError(exc) + + def close(self): + if self._entity_stack: + # If we are completing an external entity, do nothing here + return + self.feed("", isFinal = 1) + self._cont_handler.endDocument() + self._parsing = 0 + # break cycle created by expat handlers pointing to our methods + self._parser = None + + def _reset_cont_handler(self): + self._parser.ProcessingInstructionHandler = \ + self._cont_handler.processingInstruction + self._parser.CharacterDataHandler = self._cont_handler.characters + + def _reset_lex_handler_prop(self): + lex = self._lex_handler_prop + parser = self._parser + if lex is None: + parser.CommentHandler = None + parser.StartCdataSectionHandler = None + parser.EndCdataSectionHandler = None + parser.StartDoctypeDeclHandler = None + parser.EndDoctypeDeclHandler = None + else: + parser.CommentHandler = lex.comment + parser.StartCdataSectionHandler = lex.startCDATA + parser.EndCdataSectionHandler = lex.endCDATA + parser.StartDoctypeDeclHandler = self.start_doctype_decl + parser.EndDoctypeDeclHandler = lex.endDTD + + def reset(self): + if self._namespaces: + self._parser = expat.ParserCreate(self._source.getEncoding(), " ", + intern=self._interning) + self._parser.namespace_prefixes = 1 + self._parser.StartElementHandler = self.start_element_ns + self._parser.EndElementHandler = self.end_element_ns + else: + self._parser = expat.ParserCreate(self._source.getEncoding(), + intern = self._interning) + self._parser.StartElementHandler = self.start_element + self._parser.EndElementHandler = self.end_element + + self._reset_cont_handler() + self._parser.UnparsedEntityDeclHandler = self.unparsed_entity_decl + self._parser.NotationDeclHandler = self.notation_decl + self._parser.StartNamespaceDeclHandler = self.start_namespace_decl + self._parser.EndNamespaceDeclHandler = self.end_namespace_decl + + self._decl_handler_prop = None + if self._lex_handler_prop: + self._reset_lex_handler_prop() +# self._parser.DefaultHandler = +# self._parser.DefaultHandlerExpand = +# self._parser.NotStandaloneHandler = + self._parser.ExternalEntityRefHandler = self.external_entity_ref + try: + self._parser.SkippedEntityHandler = self.skipped_entity_handler + except AttributeError: + # This pyexpat does not support SkippedEntity + pass + self._parser.SetParamEntityParsing( + expat.XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE) + + self._parsing = 0 + self._entity_stack = [] + + # Locator methods + + def getColumnNumber(self): + if self._parser is None: + return None + return self._parser.ErrorColumnNumber + + def getLineNumber(self): + if self._parser is None: + return 1 + return self._parser.ErrorLineNumber + + def getPublicId(self): + return self._source.getPublicId() + + def getSystemId(self): + return self._source.getSystemId() + + # event handlers + def start_element(self, name, attrs): + self._cont_handler.startElement(name, AttributesImpl(attrs)) + + def end_element(self, name): + self._cont_handler.endElement(name) + + def start_element_ns(self, name, attrs): + pair = name.split() + if len(pair) == 1: + # no namespace + pair = (None, name) + elif len(pair) == 3: + pair = pair[0], pair[1] + else: + # default namespace + pair = tuple(pair) + + newattrs = {} + qnames = {} + for (aname, value) in attrs.items(): + parts = aname.split() + length = len(parts) + if length == 1: + # no namespace + qname = aname + apair = (None, aname) + elif length == 3: + qname = "%s:%s" % (parts[2], parts[1]) + apair = parts[0], parts[1] + else: + # default namespace + qname = parts[1] + apair = tuple(parts) + + newattrs[apair] = value + qnames[apair] = qname + + self._cont_handler.startElementNS(pair, None, + AttributesNSImpl(newattrs, qnames)) + + def end_element_ns(self, name): + pair = name.split() + if len(pair) == 1: + pair = (None, name) + elif len(pair) == 3: + pair = pair[0], pair[1] + else: + pair = tuple(pair) + + self._cont_handler.endElementNS(pair, None) + + # this is not used (call directly to ContentHandler) + def processing_instruction(self, target, data): + self._cont_handler.processingInstruction(target, data) + + # this is not used (call directly to ContentHandler) + def character_data(self, data): + self._cont_handler.characters(data) + + def start_namespace_decl(self, prefix, uri): + self._cont_handler.startPrefixMapping(prefix, uri) + + def end_namespace_decl(self, prefix): + self._cont_handler.endPrefixMapping(prefix) + + def start_doctype_decl(self, name, sysid, pubid, has_internal_subset): + self._lex_handler_prop.startDTD(name, pubid, sysid) + + def unparsed_entity_decl(self, name, base, sysid, pubid, notation_name): + self._dtd_handler.unparsedEntityDecl(name, pubid, sysid, notation_name) + + def notation_decl(self, name, base, sysid, pubid): + self._dtd_handler.notationDecl(name, pubid, sysid) + + def external_entity_ref(self, context, base, sysid, pubid): + if not self._external_ges: + return 1 + + source = self._ent_handler.resolveEntity(pubid, sysid) + source = saxutils.prepare_input_source(source, + self._source.getSystemId() or + "") + + self._entity_stack.append((self._parser, self._source)) + self._parser = self._parser.ExternalEntityParserCreate(context) + self._source = source + + try: + xmlreader.IncrementalParser.parse(self, source) + except: + return 0 # FIXME: save error info here? + + (self._parser, self._source) = self._entity_stack[-1] + del self._entity_stack[-1] + return 1 + + def skipped_entity_handler(self, name, is_pe): + if is_pe: + # The SAX spec requires to report skipped PEs with a '%' + name = '%'+name + self._cont_handler.skippedEntity(name) + +# --- + +def create_parser(*args, **kwargs): + return ExpatParser(*args, **kwargs) + +# --- + +if __name__ == "__main__": + import xml.sax + p = create_parser() + p.setContentHandler(xml.sax.XMLGenerator()) + p.setErrorHandler(xml.sax.ErrorHandler()) + p.parse("../../../hamlet.xml") Added: pypy/dist/pypy/lib/xml/sax/handler.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/sax/handler.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,342 @@ +""" +This module contains the core classes of version 2.0 of SAX for Python. +This file provides only default classes with absolutely minimum +functionality, from which drivers and applications can be subclassed. + +Many of these classes are empty and are included only as documentation +of the interfaces. + +$Id: handler.py 35816 2004-05-06 03:47:48Z fdrake $ +""" + +version = '2.0beta' + +#============================================================================ +# +# HANDLER INTERFACES +# +#============================================================================ + +# ===== ERRORHANDLER ===== + +class ErrorHandler: + """Basic interface for SAX error handlers. + + If you create an object that implements this interface, then + register the object with your XMLReader, the parser will call the + methods in your object to report all warnings and errors. There + are three levels of errors available: warnings, (possibly) + recoverable errors, and unrecoverable errors. All methods take a + SAXParseException as the only parameter.""" + + def error(self, exception): + "Handle a recoverable error." + raise exception + + def fatalError(self, exception): + "Handle a non-recoverable error." + raise exception + + def warning(self, exception): + "Handle a warning." + print exception + + +# ===== CONTENTHANDLER ===== + +class ContentHandler: + """Interface for receiving logical document content events. + + This is the main callback interface in SAX, and the one most + important to applications. The order of events in this interface + mirrors the order of the information in the document.""" + + def __init__(self): + self._locator = None + + def setDocumentLocator(self, locator): + """Called by the parser to give the application a locator for + locating the origin of document events. + + SAX parsers are strongly encouraged (though not absolutely + required) to supply a locator: if it does so, it must supply + the locator to the application by invoking this method before + invoking any of the other methods in the DocumentHandler + interface. + + The locator allows the application to determine the end + position of any document-related event, even if the parser is + not reporting an error. Typically, the application will use + this information for reporting its own errors (such as + character content that does not match an application's + business rules). The information returned by the locator is + probably not sufficient for use with a search engine. + + Note that the locator will return correct information only + during the invocation of the events in this interface. The + application should not attempt to use it at any other time.""" + self._locator = locator + + def startDocument(self): + """Receive notification of the beginning of a document. + + The SAX parser will invoke this method only once, before any + other methods in this interface or in DTDHandler (except for + setDocumentLocator).""" + + def endDocument(self): + """Receive notification of the end of a document. + + The SAX parser will invoke this method only once, and it will + be the last method invoked during the parse. The parser shall + not invoke this method until it has either abandoned parsing + (because of an unrecoverable error) or reached the end of + input.""" + + def startPrefixMapping(self, prefix, uri): + """Begin the scope of a prefix-URI Namespace mapping. + + The information from this event is not necessary for normal + Namespace processing: the SAX XML reader will automatically + replace prefixes for element and attribute names when the + http://xml.org/sax/features/namespaces feature is true (the + default). + + There are cases, however, when applications need to use + prefixes in character data or in attribute values, where they + cannot safely be expanded automatically; the + start/endPrefixMapping event supplies the information to the + application to expand prefixes in those contexts itself, if + necessary. + + Note that start/endPrefixMapping events are not guaranteed to + be properly nested relative to each-other: all + startPrefixMapping events will occur before the corresponding + startElement event, and all endPrefixMapping events will occur + after the corresponding endElement event, but their order is + not guaranteed.""" + + def endPrefixMapping(self, prefix): + """End the scope of a prefix-URI mapping. + + See startPrefixMapping for details. This event will always + occur after the corresponding endElement event, but the order + of endPrefixMapping events is not otherwise guaranteed.""" + + def startElement(self, name, attrs): + """Signals the start of an element in non-namespace mode. + + The name parameter contains the raw XML 1.0 name of the + element type as a string and the attrs parameter holds an + instance of the Attributes class containing the attributes of + the element.""" + + def endElement(self, name): + """Signals the end of an element in non-namespace mode. + + The name parameter contains the name of the element type, just + as with the startElement event.""" + + def startElementNS(self, name, qname, attrs): + """Signals the start of an element in namespace mode. + + The name parameter contains the name of the element type as a + (uri, localname) tuple, the qname parameter the raw XML 1.0 + name used in the source document, and the attrs parameter + holds an instance of the Attributes class containing the + attributes of the element. + + The uri part of the name tuple is None for elements which have + no namespace.""" + + def endElementNS(self, name, qname): + """Signals the end of an element in namespace mode. + + The name parameter contains the name of the element type, just + as with the startElementNS event.""" + + def characters(self, content): + """Receive notification of character data. + + The Parser will call this method to report each chunk of + character data. SAX parsers may return all contiguous + character data in a single chunk, or they may split it into + several chunks; however, all of the characters in any single + event must come from the same external entity so that the + Locator provides useful information.""" + + def ignorableWhitespace(self, whitespace): + """Receive notification of ignorable whitespace in element content. + + Validating Parsers must use this method to report each chunk + of ignorable whitespace (see the W3C XML 1.0 recommendation, + section 2.10): non-validating parsers may also use this method + if they are capable of parsing and using content models. + + SAX parsers may return all contiguous whitespace in a single + chunk, or they may split it into several chunks; however, all + of the characters in any single event must come from the same + external entity, so that the Locator provides useful + information.""" + + def processingInstruction(self, target, data): + """Receive notification of a processing instruction. + + The Parser will invoke this method once for each processing + instruction found: note that processing instructions may occur + before or after the main document element. + + A SAX parser should never report an XML declaration (XML 1.0, + section 2.8) or a text declaration (XML 1.0, section 4.3.1) + using this method.""" + + def skippedEntity(self, name): + """Receive notification of a skipped entity. + + The Parser will invoke this method once for each entity + skipped. Non-validating processors may skip entities if they + have not seen the declarations (because, for example, the + entity was declared in an external DTD subset). All processors + may skip external entities, depending on the values of the + http://xml.org/sax/features/external-general-entities and the + http://xml.org/sax/features/external-parameter-entities + properties.""" + + +# ===== DTDHandler ===== + +class DTDHandler: + """Handle DTD events. + + This interface specifies only those DTD events required for basic + parsing (unparsed entities and attributes).""" + + def notationDecl(self, name, publicId, systemId): + "Handle a notation declaration event." + + def unparsedEntityDecl(self, name, publicId, systemId, ndata): + "Handle an unparsed entity declaration event." + + +# ===== ENTITYRESOLVER ===== + +class EntityResolver: + """Basic interface for resolving entities. If you create an object + implementing this interface, then register the object with your + Parser, the parser will call the method in your object to + resolve all external entities. Note that DefaultHandler implements + this interface with the default behaviour.""" + + def resolveEntity(self, publicId, systemId): + """Resolve the system identifier of an entity and return either + the system identifier to read from as a string, or an InputSource + to read from.""" + return systemId + + +#============================================================================ +# +# CORE FEATURES +# +#============================================================================ + +feature_namespaces = "http://xml.org/sax/features/namespaces" +# true: Perform Namespace processing (default). +# false: Optionally do not perform Namespace processing +# (implies namespace-prefixes). +# access: (parsing) read-only; (not parsing) read/write + +feature_namespace_prefixes = "http://xml.org/sax/features/namespace-prefixes" +# true: Report the original prefixed names and attributes used for Namespace +# declarations. +# false: Do not report attributes used for Namespace declarations, and +# optionally do not report original prefixed names (default). +# access: (parsing) read-only; (not parsing) read/write + +feature_string_interning = "http://xml.org/sax/features/string-interning" +# true: All element names, prefixes, attribute names, Namespace URIs, and +# local names are interned using the built-in intern function. +# false: Names are not necessarily interned, although they may be (default). +# access: (parsing) read-only; (not parsing) read/write + +feature_validation = "http://xml.org/sax/features/validation" +# true: Report all validation errors (implies external-general-entities and +# external-parameter-entities). +# false: Do not report validation errors. +# access: (parsing) read-only; (not parsing) read/write + +feature_external_ges = "http://xml.org/sax/features/external-general-entities" +# true: Include all external general (text) entities. +# false: Do not include external general entities. +# access: (parsing) read-only; (not parsing) read/write + +feature_external_pes = "http://xml.org/sax/features/external-parameter-entities" +# true: Include all external parameter entities, including the external +# DTD subset. +# false: Do not include any external parameter entities, even the external +# DTD subset. +# access: (parsing) read-only; (not parsing) read/write + +all_features = [feature_namespaces, + feature_namespace_prefixes, + feature_string_interning, + feature_validation, + feature_external_ges, + feature_external_pes] + + +#============================================================================ +# +# CORE PROPERTIES +# +#============================================================================ + +property_lexical_handler = "http://xml.org/sax/properties/lexical-handler" +# data type: xml.sax.sax2lib.LexicalHandler +# description: An optional extension handler for lexical events like comments. +# access: read/write + +property_declaration_handler = "http://xml.org/sax/properties/declaration-handler" +# data type: xml.sax.sax2lib.DeclHandler +# description: An optional extension handler for DTD-related events other +# than notations and unparsed entities. +# access: read/write + +property_dom_node = "http://xml.org/sax/properties/dom-node" +# data type: org.w3c.dom.Node +# description: When parsing, the current DOM node being visited if this is +# a DOM iterator; when not parsing, the root DOM node for +# iteration. +# access: (parsing) read-only; (not parsing) read/write + +property_xml_string = "http://xml.org/sax/properties/xml-string" +# data type: String +# description: The literal string of characters that was the source for +# the current event. +# access: read-only + +property_encoding = "http://www.python.org/sax/properties/encoding" +# data type: String +# description: The name of the encoding to assume for input data. +# access: write: set the encoding, e.g. established by a higher-level +# protocol. May change during parsing (e.g. after +# processing a META tag) +# read: return the current encoding (possibly established through +# auto-detection. +# initial value: UTF-8 +# + +property_interning_dict = "http://www.python.org/sax/properties/interning-dict" +# data type: Dictionary +# description: The dictionary used to intern common strings in the document +# access: write: Request that the parser uses a specific dictionary, to +# allow interning across different documents +# read: return the current interning dictionary, or None +# + +all_properties = [property_lexical_handler, + property_dom_node, + property_declaration_handler, + property_xml_string, + property_encoding, + property_interning_dict] Added: pypy/dist/pypy/lib/xml/sax/saxutils.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/sax/saxutils.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,302 @@ +"""\ +A library of useful helper classes to the SAX classes, for the +convenience of application and driver writers. +""" + +import os, urlparse, urllib, types +import handler +import xmlreader + +try: + _StringTypes = [types.StringType, types.UnicodeType] +except AttributeError: + _StringTypes = [types.StringType] + +# See whether the xmlcharrefreplace error handler is +# supported +try: + from codecs import xmlcharrefreplace_errors + _error_handling = "xmlcharrefreplace" + del xmlcharrefreplace_errors +except ImportError: + _error_handling = "strict" + +def __dict_replace(s, d): + """Replace substrings of a string using a dictionary.""" + for key, value in d.items(): + s = s.replace(key, value) + return s + +def escape(data, entities={}): + """Escape &, <, and > in a string of data. + + You can escape other strings of data by passing a dictionary as + the optional entities parameter. The keys and values must all be + strings; each key will be replaced with its corresponding value. + """ + + # must do ampersand first + data = data.replace("&", "&") + data = data.replace(">", ">") + data = data.replace("<", "<") + if entities: + data = __dict_replace(data, entities) + return data + +def unescape(data, entities={}): + """Unescape &, <, and > in a string of data. + + You can unescape other strings of data by passing a dictionary as + the optional entities parameter. The keys and values must all be + strings; each key will be replaced with its corresponding value. + """ + data = data.replace("<", "<") + data = data.replace(">", ">") + if entities: + data = __dict_replace(data, entities) + # must do ampersand last + return data.replace("&", "&") + +def quoteattr(data, entities={}): + """Escape and quote an attribute value. + + Escape &, <, and > in a string of data, then quote it for use as + an attribute value. The \" character will be escaped as well, if + necessary. + + You can escape other strings of data by passing a dictionary as + the optional entities parameter. The keys and values must all be + strings; each key will be replaced with its corresponding value. + """ + entities = entities.copy() + entities.update({'\n': ' ', '\r': ' ', '\t':' '}) + data = escape(data, entities) + if '"' in data: + if "'" in data: + data = '"%s"' % data.replace('"', """) + else: + data = "'%s'" % data + else: + data = '"%s"' % data + return data + + +class XMLGenerator(handler.ContentHandler): + + def __init__(self, out=None, encoding="iso-8859-1"): + if out is None: + import sys + out = sys.stdout + handler.ContentHandler.__init__(self) + self._out = out + self._ns_contexts = [{}] # contains uri -> prefix dicts + self._current_context = self._ns_contexts[-1] + self._undeclared_ns_maps = [] + self._encoding = encoding + + def _write(self, text): + if isinstance(text, str): + self._out.write(text) + else: + self._out.write(text.encode(self._encoding, _error_handling)) + + def _qname(self, name): + """Builds a qualified name from a (ns_url, localname) pair""" + if name[0]: + # The name is in a non-empty namespace + prefix = self._current_context[name[0]] + if prefix: + # If it is not the default namespace, prepend the prefix + return prefix + ":" + name[1] + # Return the unqualified name + return name[1] + + # ContentHandler methods + + def startDocument(self): + self._write('\n' % + self._encoding) + + def startPrefixMapping(self, prefix, uri): + self._ns_contexts.append(self._current_context.copy()) + self._current_context[uri] = prefix + self._undeclared_ns_maps.append((prefix, uri)) + + def endPrefixMapping(self, prefix): + self._current_context = self._ns_contexts[-1] + del self._ns_contexts[-1] + + def startElement(self, name, attrs): + self._write('<' + name) + for (name, value) in attrs.items(): + self._write(' %s=%s' % (name, quoteattr(value))) + self._write('>') + + def endElement(self, name): + self._write('' % name) + + def startElementNS(self, name, qname, attrs): + self._write('<' + self._qname(name)) + + for prefix, uri in self._undeclared_ns_maps: + if prefix: + self._out.write(' xmlns:%s="%s"' % (prefix, uri)) + else: + self._out.write(' xmlns="%s"' % uri) + self._undeclared_ns_maps = [] + + for (name, value) in attrs.items(): + self._write(' %s=%s' % (self._qname(name), quoteattr(value))) + self._write('>') + + def endElementNS(self, name, qname): + self._write('' % self._qname(name)) + + def characters(self, content): + self._write(escape(content)) + + def ignorableWhitespace(self, content): + self._write(content) + + def processingInstruction(self, target, data): + self._write('' % (target, data)) + + +class XMLFilterBase(xmlreader.XMLReader): + """This class is designed to sit between an XMLReader and the + client application's event handlers. By default, it does nothing + but pass requests up to the reader and events on to the handlers + unmodified, but subclasses can override specific methods to modify + the event stream or the configuration requests as they pass + through.""" + + def __init__(self, parent = None): + xmlreader.XMLReader.__init__(self) + self._parent = parent + + # ErrorHandler methods + + def error(self, exception): + self._err_handler.error(exception) + + def fatalError(self, exception): + self._err_handler.fatalError(exception) + + def warning(self, exception): + self._err_handler.warning(exception) + + # ContentHandler methods + + def setDocumentLocator(self, locator): + self._cont_handler.setDocumentLocator(locator) + + def startDocument(self): + self._cont_handler.startDocument() + + def endDocument(self): + self._cont_handler.endDocument() + + def startPrefixMapping(self, prefix, uri): + self._cont_handler.startPrefixMapping(prefix, uri) + + def endPrefixMapping(self, prefix): + self._cont_handler.endPrefixMapping(prefix) + + def startElement(self, name, attrs): + self._cont_handler.startElement(name, attrs) + + def endElement(self, name): + self._cont_handler.endElement(name) + + def startElementNS(self, name, qname, attrs): + self._cont_handler.startElementNS(name, qname, attrs) + + def endElementNS(self, name, qname): + self._cont_handler.endElementNS(name, qname) + + def characters(self, content): + self._cont_handler.characters(content) + + def ignorableWhitespace(self, chars): + self._cont_handler.ignorableWhitespace(chars) + + def processingInstruction(self, target, data): + self._cont_handler.processingInstruction(target, data) + + def skippedEntity(self, name): + self._cont_handler.skippedEntity(name) + + # DTDHandler methods + + def notationDecl(self, name, publicId, systemId): + self._dtd_handler.notationDecl(name, publicId, systemId) + + def unparsedEntityDecl(self, name, publicId, systemId, ndata): + self._dtd_handler.unparsedEntityDecl(name, publicId, systemId, ndata) + + # EntityResolver methods + + def resolveEntity(self, publicId, systemId): + return self._ent_handler.resolveEntity(publicId, systemId) + + # XMLReader methods + + def parse(self, source): + self._parent.setContentHandler(self) + self._parent.setErrorHandler(self) + self._parent.setEntityResolver(self) + self._parent.setDTDHandler(self) + self._parent.parse(source) + + def setLocale(self, locale): + self._parent.setLocale(locale) + + def getFeature(self, name): + return self._parent.getFeature(name) + + def setFeature(self, name, state): + self._parent.setFeature(name, state) + + def getProperty(self, name): + return self._parent.getProperty(name) + + def setProperty(self, name, value): + self._parent.setProperty(name, value) + + # XMLFilter methods + + def getParent(self): + return self._parent + + def setParent(self, parent): + self._parent = parent + +# --- Utility functions + +def prepare_input_source(source, base = ""): + """This function takes an InputSource and an optional base URL and + returns a fully resolved InputSource object ready for reading.""" + + if type(source) in _StringTypes: + source = xmlreader.InputSource(source) + elif hasattr(source, "read"): + f = source + source = xmlreader.InputSource() + source.setByteStream(f) + if hasattr(f, "name"): + source.setSystemId(f.name) + + if source.getByteStream() is None: + sysid = source.getSystemId() + basehead = os.path.dirname(os.path.normpath(base)) + sysidfilename = os.path.join(basehead, sysid) + if os.path.isfile(sysidfilename): + source.setSystemId(sysidfilename) + f = open(sysidfilename, "rb") + else: + source.setSystemId(urlparse.urljoin(base, sysid)) + f = urllib.urlopen(source.getSystemId()) + + source.setByteStream(f) + + return source Added: pypy/dist/pypy/lib/xml/sax/xmlreader.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/xml/sax/xmlreader.py Fri Jul 11 11:14:56 2008 @@ -0,0 +1,381 @@ +"""An XML Reader is the SAX 2 name for an XML parser. XML Parsers +should be based on this code. """ + +import handler + +from _exceptions import SAXNotSupportedException, SAXNotRecognizedException + + +# ===== XMLREADER ===== + +class XMLReader: + """Interface for reading an XML document using callbacks. + + XMLReader is the interface that an XML parser's SAX2 driver must + implement. This interface allows an application to set and query + features and properties in the parser, to register event handlers + for document processing, and to initiate a document parse. + + All SAX interfaces are assumed to be synchronous: the parse + methods must not return until parsing is complete, and readers + must wait for an event-handler callback to return before reporting + the next event.""" + + def __init__(self): + self._cont_handler = handler.ContentHandler() + self._dtd_handler = handler.DTDHandler() + self._ent_handler = handler.EntityResolver() + self._err_handler = handler.ErrorHandler() + + def parse(self, source): + "Parse an XML document from a system identifier or an InputSource." + raise NotImplementedError("This method must be implemented!") + + def getContentHandler(self): + "Returns the current ContentHandler." + return self._cont_handler + + def setContentHandler(self, handler): + "Registers a new object to receive document content events." + self._cont_handler = handler + + def getDTDHandler(self): + "Returns the current DTD handler." + return self._dtd_handler + + def setDTDHandler(self, handler): + "Register an object to receive basic DTD-related events." + self._dtd_handler = handler + + def getEntityResolver(self): + "Returns the current EntityResolver." + return self._ent_handler + + def setEntityResolver(self, resolver): + "Register an object to resolve external entities." + self._ent_handler = resolver + + def getErrorHandler(self): + "Returns the current ErrorHandler." + return self._err_handler + + def setErrorHandler(self, handler): + "Register an object to receive error-message events." + self._err_handler = handler + + def setLocale(self, locale): + """Allow an application to set the locale for errors and warnings. + + SAX parsers are not required to provide localization for errors + and warnings; if they cannot support the requested locale, + however, they must throw a SAX exception. Applications may + request a locale change in the middle of a parse.""" + raise SAXNotSupportedException("Locale support not implemented") + + def getFeature(self, name): + "Looks up and returns the state of a SAX2 feature." + raise SAXNotRecognizedException("Feature '%s' not recognized" % name) + + def setFeature(self, name, state): + "Sets the state of a SAX2 feature." + raise SAXNotRecognizedException("Feature '%s' not recognized" % name) + + def getProperty(self, name): + "Looks up and returns the value of a SAX2 property." + raise SAXNotRecognizedException("Property '%s' not recognized" % name) + + def setProperty(self, name, value): + "Sets the value of a SAX2 property." + raise SAXNotRecognizedException("Property '%s' not recognized" % name) + +class IncrementalParser(XMLReader): + """This interface adds three extra methods to the XMLReader + interface that allow XML parsers to support incremental + parsing. Support for this interface is optional, since not all + underlying XML parsers support this functionality. + + When the parser is instantiated it is ready to begin accepting + data from the feed method immediately. After parsing has been + finished with a call to close the reset method must be called to + make the parser ready to accept new data, either from feed or + using the parse method. + + Note that these methods must _not_ be called during parsing, that + is, after parse has been called and before it returns. + + By default, the class also implements the parse method of the XMLReader + interface using the feed, close and reset methods of the + IncrementalParser interface as a convenience to SAX 2.0 driver + writers.""" + + def __init__(self, bufsize=2**16): + self._bufsize = bufsize + XMLReader.__init__(self) + + def parse(self, source): + import saxutils + source = saxutils.prepare_input_source(source) + + self.prepareParser(source) + file = source.getByteStream() + buffer = file.read(self._bufsize) + while buffer != "": + self.feed(buffer) + buffer = file.read(self._bufsize) + self.close() + + def feed(self, data): + """This method gives the raw XML data in the data parameter to + the parser and makes it parse the data, emitting the + corresponding events. It is allowed for XML constructs to be + split across several calls to feed. + + feed may raise SAXException.""" + raise NotImplementedError("This method must be implemented!") + + def prepareParser(self, source): + """This method is called by the parse implementation to allow + the SAX 2.0 driver to prepare itself for parsing.""" + raise NotImplementedError("prepareParser must be overridden!") + + def close(self): + """This method is called when the entire XML document has been + passed to the parser through the feed method, to notify the + parser that there are no more data. This allows the parser to + do the final checks on the document and empty the internal + data buffer. + + The parser will not be ready to parse another document until + the reset method has been called. + + close may raise SAXException.""" + raise NotImplementedError("This method must be implemented!") + + def reset(self): + """This method is called after close has been called to reset + the parser so that it is ready to parse new documents. The + results of calling parse or feed after close without calling + reset are undefined.""" + raise NotImplementedError("This method must be implemented!") + +# ===== LOCATOR ===== + +class Locator: + """Interface for associating a SAX event with a document + location. A locator object will return valid results only during + calls to DocumentHandler methods; at any other time, the + results are unpredictable.""" + + def getColumnNumber(self): + "Return the column number where the current event ends." + return -1 + + def getLineNumber(self): + "Return the line number where the current event ends." + return -1 + + def getPublicId(self): + "Return the public identifier for the current event." + return None + + def getSystemId(self): + "Return the system identifier for the current event." + return None + +# ===== INPUTSOURCE ===== + +class InputSource: + """Encapsulation of the information needed by the XMLReader to + read entities. + + This class may include information about the public identifier, + system identifier, byte stream (possibly with character encoding + information) and/or the character stream of an entity. + + Applications will create objects of this class for use in the + XMLReader.parse method and for returning from + EntityResolver.resolveEntity. + + An InputSource belongs to the application, the XMLReader is not + allowed to modify InputSource objects passed to it from the + application, although it may make copies and modify those.""" + + def __init__(self, system_id = None): + self.__system_id = system_id + self.__public_id = None + self.__encoding = None + self.__bytefile = None + self.__charfile = None + + def setPublicId(self, public_id): + "Sets the public identifier of this InputSource." + self.__public_id = public_id + + def getPublicId(self): + "Returns the public identifier of this InputSource." + return self.__public_id + + def setSystemId(self, system_id): + "Sets the system identifier of this InputSource." + self.__system_id = system_id + + def getSystemId(self): + "Returns the system identifier of this InputSource." + return self.__system_id + + def setEncoding(self, encoding): + """Sets the character encoding of this InputSource. + + The encoding must be a string acceptable for an XML encoding + declaration (see section 4.3.3 of the XML recommendation). + + The encoding attribute of the InputSource is ignored if the + InputSource also contains a character stream.""" + self.__encoding = encoding + + def getEncoding(self): + "Get the character encoding of this InputSource." + return self.__encoding + + def setByteStream(self, bytefile): + """Set the byte stream (a Python file-like object which does + not perform byte-to-character conversion) for this input + source. + + The SAX parser will ignore this if there is also a character + stream specified, but it will use a byte stream in preference + to opening a URI connection itself. + + If the application knows the character encoding of the byte + stream, it should set it with the setEncoding method.""" + self.__bytefile = bytefile + + def getByteStream(self): + """Get the byte stream for this input source. + + The getEncoding method will return the character encoding for + this byte stream, or None if unknown.""" + return self.__bytefile + + def setCharacterStream(self, charfile): + """Set the character stream for this input source. (The stream + must be a Python 2.0 Unicode-wrapped file-like that performs + conversion to Unicode strings.) + + If there is a character stream specified, the SAX parser will + ignore any byte stream and will not attempt to open a URI + connection to the system identifier.""" + self.__charfile = charfile + + def getCharacterStream(self): + "Get the character stream for this input source." + return self.__charfile + +# ===== ATTRIBUTESIMPL ===== + +class AttributesImpl: + + def __init__(self, attrs): + """Non-NS-aware implementation. + + attrs should be of the form {name : value}.""" + self._attrs = attrs + + def getLength(self): + return len(self._attrs) + + def getType(self, name): + return "CDATA" + + def getValue(self, name): + return self._attrs[name] + + def getValueByQName(self, name): + return self._attrs[name] + + def getNameByQName(self, name): + if not self._attrs.has_key(name): + raise KeyError, name + return name + + def getQNameByName(self, name): + if not self._attrs.has_key(name): + raise KeyError, name + return name + + def getNames(self): + return self._attrs.keys() + + def getQNames(self): + return self._attrs.keys() + + def __len__(self): + return len(self._attrs) + + def __getitem__(self, name): + return self._attrs[name] + + def keys(self): + return self._attrs.keys() + + def has_key(self, name): + return self._attrs.has_key(name) + + def __contains__(self, name): + return self._attrs.has_key(name) + + def get(self, name, alternative=None): + return self._attrs.get(name, alternative) + + def copy(self): + return self.__class__(self._attrs) + + def items(self): + return self._attrs.items() + + def values(self): + return self._attrs.values() + +# ===== ATTRIBUTESNSIMPL ===== + +class AttributesNSImpl(AttributesImpl): + + def __init__(self, attrs, qnames): + """NS-aware implementation. + + attrs should be of the form {(ns_uri, lname): value, ...}. + qnames of the form {(ns_uri, lname): qname, ...}.""" + self._attrs = attrs + self._qnames = qnames + + def getValueByQName(self, name): + for (nsname, qname) in self._qnames.items(): + if qname == name: + return self._attrs[nsname] + + raise KeyError, name + + def getNameByQName(self, name): + for (nsname, qname) in self._qnames.items(): + if qname == name: + return nsname + + raise KeyError, name + + def getQNameByName(self, name): + return self._qnames[name] + + def getQNames(self): + return self._qnames.values() + + def copy(self): + return self.__class__(self._attrs, self._qnames) + + +def _test(): + XMLReader() + IncrementalParser() + Locator() + +if __name__ == "__main__": + _test() From stephan at codespeak.net Fri Jul 11 11:20:38 2008 From: stephan at codespeak.net (stephan at codespeak.net) Date: Fri, 11 Jul 2008 11:20:38 +0200 (CEST) Subject: [pypy-svn] r56440 - in pypy/branch/builtin-profiling/pypy/interpreter: . test Message-ID: <20080711092038.874AE398003@codespeak.net> Author: stephan Date: Fri Jul 11 11:20:38 2008 New Revision: 56440 Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py pypy/branch/builtin-profiling/pypy/interpreter/gateway.py pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Log: (stephan, anto) first (non working) checkin to support profiling of builtin functions. (much needs to be changed) Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py Fri Jul 11 11:20:38 2008 @@ -104,6 +104,31 @@ space.setitem(w_globals, w_key, w_value) return w_globals + def c_call_llprofile(self, w_func): + "Profile the call of a builtin function" + if self.profilefunc is not None: + self._llprofile('c_call', w_func) + + def c_return_llprofile(self, w_retval): + "Profile the return from a builtin function" + if self.profilefunc is not None: + self._llprofile('c_return', w_retval) + + def c_exception_llprofile(self, operationerr): + "Profile function called upon OperationError." + if self.profilefunc is not None: + self._llprofile('c_exception', operationerr) + + def _llprofile(self, event, w_arg): + fr = self.framestack.items + space = self.space + w_callback = self.profilefunc + if w_callback is not None: + frame = None + if fr: + frame = fr[0] + self.profilefunc(space, self.w_profilefuncarg, frame, event, w_arg) + def call_trace(self, frame): "Trace the call of a function" if self.w_tracefunc is not None or self.profilefunc is not None: @@ -182,7 +207,7 @@ space = self.space # Tracing cases - if event == 'call': + if event in ['call']: w_callback = self.w_tracefunc else: w_callback = frame.w_f_trace Modified: pypy/branch/builtin-profiling/pypy/interpreter/gateway.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/gateway.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/gateway.py Fri Jul 11 11:20:38 2008 @@ -480,10 +480,13 @@ def funcrun(self, func, args): space = func.space + ctx = space.getexecutioncontext() activation = self.activation scope_w = args.parse(func.name, self.sig, func.defs_w) try: + ctx.c_call_llprofile(activation._run) w_result = activation._run(space, scope_w) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) @@ -509,8 +512,11 @@ def funcrun(self, func, args): space = func.space + ctx = space.getexecutioncontext() try: + ctx.c_call_llprofile(self.func__args__) w_result = self.func__args__(space, args) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -533,13 +539,16 @@ def funcrun(self, func, args): space = func.space + ctx = space.getexecutioncontext() try: w_obj, newargs = args.popfirst() except IndexError: return BuiltinCode.funcrun(self, func, args) else: try: + ctx.c_call_llprofile(self.func__args__) w_result = self.func__args__(space, w_obj, newargs) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -562,7 +571,10 @@ def fastcall_0(self, space, w_func): self = hint(self, deepfreeze=True) try: + ctx = space.getexecutioncontext() + ctx.c_call_llprofile(self.fastfunc_0) w_result = self.fastfunc_0(space) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -578,7 +590,10 @@ def fastcall_1(self, space, w_func, w1): self = hint(self, deepfreeze=True) try: + ctx = space.getexecutioncontext() + ctx.c_call_llprofile(self.fastfunc_1) w_result = self.fastfunc_1(space, w1) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -601,7 +616,10 @@ def fastcall_2(self, space, w_func, w1, w2): self = hint(self, deepfreeze=True) try: + ctx = space.getexecutioncontext() + ctx.c_call_llprofile(self.fastfunc_2) w_result = self.fastfunc_2(space, w1, w2) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -621,10 +639,13 @@ return w_result class BuiltinCode3(BuiltinCode): - def fastcall_3(self, space, func, w1, w2, w3): + def fastcall_3(self, space, w_func, w1, w2, w3): self = hint(self, deepfreeze=True) try: + ctx = space.getexecutioncontext() + ctx.c_call_llprofile(self.fastfunc_3) w_result = self.fastfunc_3(space, w1, w2, w3) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -644,10 +665,13 @@ return w_result class BuiltinCode4(BuiltinCode): - def fastcall_4(self, space, func, w1, w2, w3, w4): + def fastcall_4(self, space, w_func, w1, w2, w3, w4): self = hint(self, deepfreeze=True) try: + ctx = space.getexecutioncontext() + ctx.c_call_llprofile(self.fastfunc_4) w_result = self.fastfunc_4(space, w1, w2, w3, w4) + ctx.c_return_llprofile(w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: Modified: pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Fri Jul 11 11:20:38 2008 @@ -62,12 +62,11 @@ assert space.sys.checkinterval / 10 < i < space.sys.checkinterval * 3 def test_llprofile(self): - py.test.skip("not working yet") l = [] def profile_func(space, w_arg, frame, event, w_aarg): assert w_arg is space.w_None - l.append(event) + l.append((event, frame, w_aarg)) space = self.space space.getexecutioncontext().setllprofile(profile_func, space.w_None) @@ -76,5 +75,7 @@ l.append(3) """) space.getexecutioncontext().setllprofile(None, None) + from pprint import pprint + pprint([(x, y, z) for x,y,z in l if x in ('call','c_call')]) assert l == ['call', 'return', 'call', 'c_call', 'c_return', 'return'] From jlg at codespeak.net Fri Jul 11 12:42:39 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Fri, 11 Jul 2008 12:42:39 +0200 (CEST) Subject: [pypy-svn] r56441 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711104239.931D549801C@codespeak.net> Author: jlg Date: Fri Jul 11 12:42:37 2008 New Revision: 56441 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - imap implemented Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Fri Jul 11 12:42:37 2008 @@ -30,6 +30,7 @@ 'dropwhile' : 'interp_itertools.W_DropWhile', 'ifilter' : 'interp_itertools.W_IFilter', 'ifilterfalse' : 'interp_itertools.W_IFilterFalse', + 'imap' : 'interp_itertools.W_IMap', 'islice' : 'interp_itertools.W_ISlice', 'repeat' : 'interp_itertools.W_Repeat', 'takewhile' : 'interp_itertools.W_TakeWhile', Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 12:42:37 2008 @@ -215,8 +215,9 @@ def __init__(self, space, w_predicate, w_iterable): self.space = space if space.is_w(w_predicate, space.w_None): - self.w_predicate = space.w_bool + self.no_predicate = True else: + self.no_predicate = False self.w_predicate = w_predicate self.iterable = space.iter(w_iterable) @@ -226,8 +227,11 @@ def next_w(self): while True: w_obj = self.space.next(self.iterable) # may raise w_StopIteration - w_pred = self.space.call_function(self.w_predicate, w_obj) - pred = self.space.is_true(w_pred) + if self.no_predicate: + pred = self.space.is_true(w_obj) + else: + w_pred = self.space.call_function(self.w_predicate, w_obj) + pred = self.space.is_true(w_pred) if pred ^ self.reverse: return w_obj @@ -303,9 +307,11 @@ raise OperationError(space.w_TypeError, space.wrap("islice() takes at most 4 arguments (" + str(num_args) + " given)")) if space.is_w(w_stop, space.w_None): - stop = None + stop = 0 + self.stoppable = False else: stop = space.int_w(w_stop) + self.stoppable = True if num_args == 2: step = space.int_w(args_w[1]) @@ -314,7 +320,7 @@ if start < 0: raise OperationError(space.w_ValueError, space.wrap("Indicies for islice() must be non-negative integers.")) - if stop is not None and stop < 0: + if self.stoppable and stop < 0: raise OperationError(space.w_ValueError, space.wrap("Stop argument must be a non-negative integer or None.")) if step < 1: raise OperationError(space.w_ValueError, space.wrap("Step must be one or lager for islice().")) @@ -327,7 +333,7 @@ return self.space.wrap(self) def next_w(self): - if self.stop is not None and self.stop <= 0: + if self.stoppable and self.stop <= 0: raise OperationError(self.space.w_StopIteration, self.space.w_None) if self.start >= 0: @@ -339,11 +345,11 @@ while skip > 0: self.space.next(self.iterable) skip -= 1 - if self.stop is not None: + if self.stoppable: self.stop -= 1 w_obj = self.space.next(self.iterable) - if self.stop is not None: + if self.stoppable: self.stop -= 1 return w_obj @@ -375,7 +381,8 @@ def __init__(self, space, args_w): self.space = space iterators_w = [] - for i, iterable_w in enumerate(args_w): + i = 0 + for iterable_w in args_w: try: iterator_w = space.iter(iterable_w) except OperationError, e: @@ -385,7 +392,10 @@ raise else: iterators_w.append(iterator_w) - self.iterators_w = iter(iterators_w) + + i += 1 + + self.iterators = iter(iterators_w) self.started = False def iter_w(self): @@ -394,7 +404,7 @@ def next_w(self): if not self.started: try: - self.w_it = self.iterators_w.next() + self.w_it = self.iterators.next() except StopIteration: raise OperationError(self.space.w_StopIteration, self.space.w_None) else: @@ -406,7 +416,7 @@ except OperationError, e: if e.match(self.space, self.space.w_StopIteration): try: - self.w_it = self.iterators_w.next() + self.w_it = self.iterators.next() except StopIteration: raise OperationError(self.space.w_StopIteration, self.space.w_None) else: @@ -437,3 +447,75 @@ for element in it: yield element """) + +class W_IMap(Wrappable): + + def __init__(self, space, w_fun, args_w): + self.space = space + self.identity_fun = (self.space.is_w(w_fun, space.w_None)) + self.w_fun = w_fun + + iterators_w = [] + i = 0 + for iterable_w in args_w: + try: + iterator_w = space.iter(iterable_w) + except OperationError, e: + if e.match(self.space, self.space.w_TypeError): + raise OperationError(space.w_TypeError, space.wrap("imap argument #" + str(i + 1) + " must support iteration")) + else: + raise + else: + iterators_w.append(iterator_w) + + i += 1 + + self.iterators_w = iterators_w + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if not self.iterators_w: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + w_objects = [self.space.next(w_it) for w_it in self.iterators_w] + + if self.identity_fun: + return self.space.newtuple(w_objects) + else: + return self.space.call_function(self.w_fun, *w_objects) + + +def W_IMap___new__(space, w_subtype, w_fun, args_w): + result = space.allocate_instance(W_IMap, w_subtype) + W_IMap.__init__(result, space, w_fun, args_w) + return space.wrap(result) + +W_IMap.typedef = TypeDef( + 'imap', + __new__ = interp2app(W_IMap___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, 'args_w']), + __iter__ = interp2app(W_IMap.iter_w, unwrap_spec=['self']), + next = interp2app(W_IMap.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that computes the function using arguments + from each of the iterables. If function is set to None, then + imap() returns the arguments as a tuple. Like map() but stops + when the shortest iterable is exhausted instead of filling in + None for shorter iterables. The reason for the difference is that + infinite iterator arguments are typically an error for map() + (because the output is fully evaluated) but represent a common + and useful way of supplying arguments to imap(). + + Equivalent to : + + def imap(function, *iterables): + iterables = map(iter, iterables) + while True: + args = [i.next() for i in iterables] + if function is None: + yield tuple(args) + else: + yield function(*args) + + """) + Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 12:42:37 2008 @@ -16,6 +16,7 @@ itertools.ifilterfalse(None, []), itertools.islice([], 0), itertools.chain(), + itertools.imap(None), ] for it in iterables: @@ -277,6 +278,36 @@ except TypeError, e: assert str(e) == "chain argument #%d must support iteration" % (x + 1) + def test_imap(self): + import itertools + + it = itertools.imap(None) + raises(StopIteration, it.next) + + obj_list = [object(), object(), object()] + it = itertools.imap(None, obj_list) + for x in obj_list: + assert it.next() == (x, ) + raises(StopIteration, it.next) + + it = itertools.imap(None, [1, 2, 3], [4], [5, 6]) + assert it.next() == (1, 4, 5) + raises(StopIteration, it.next) + + it = itertools.imap(None, [], [], [1], []) + raises(StopIteration, it.next) + + it = itertools.imap(str, [0, 1, 0, 1]) + for x in ['0', '1', '0', '1']: + assert it.next() == x + raises(StopIteration, it.next) + + import operator + it = itertools.imap(operator.add, [1, 2, 3], [4, 5, 6]) + for x in [5, 7, 9]: + assert it.next() == x + raises(StopIteration, it.next) + def test_docstrings(self): import itertools @@ -290,6 +321,7 @@ itertools.ifilterfalse, itertools.islice, itertools.chain, + itertools.imap, ] for method in methods: assert method.__doc__ @@ -306,6 +338,7 @@ (itertools.ifilterfalse, (None, [])), (itertools.islice, ([], 0)), (itertools.chain, ()), + (itertools.imap, (None,)), ] for cls, args in iterables: class A(cls): From arigo at codespeak.net Fri Jul 11 14:13:49 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 11 Jul 2008 14:13:49 +0200 (CEST) Subject: [pypy-svn] r56442 - pypy/dist/pypy/objspace/fake Message-ID: <20080711121349.69071398005@codespeak.net> Author: arigo Date: Fri Jul 11 14:13:48 2008 New Revision: 56442 Modified: pypy/dist/pypy/objspace/fake/objspace.py Log: Make the fake object space stricter: check that attributes don't move up to W_Object or W_Type. Modified: pypy/dist/pypy/objspace/fake/objspace.py ============================================================================== --- pypy/dist/pypy/objspace/fake/objspace.py (original) +++ pypy/dist/pypy/objspace/fake/objspace.py Fri Jul 11 14:13:48 2008 @@ -4,10 +4,10 @@ from pypy.rlib.rbigint import rbigint class W_Type(W_Root): - pass + _attrs_ = () class W_Object(W_Root): - pass + _attrs_ = () W_Object.typedef = W_Type() def make_dummy(a=W_Object(), b=W_Object()): From stephan at codespeak.net Fri Jul 11 14:15:41 2008 From: stephan at codespeak.net (stephan at codespeak.net) Date: Fri, 11 Jul 2008 14:15:41 +0200 (CEST) Subject: [pypy-svn] r56443 - in pypy/branch/builtin-profiling/pypy/interpreter: . test Message-ID: <20080711121541.AB694398005@codespeak.net> Author: stephan Date: Fri Jul 11 14:15:41 2008 New Revision: 56443 Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py pypy/branch/builtin-profiling/pypy/interpreter/gateway.py pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Log: (stephan) another not working version Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py Fri Jul 11 14:15:41 2008 @@ -104,20 +104,23 @@ space.setitem(w_globals, w_key, w_value) return w_globals - def c_call_llprofile(self, w_func): + def c_call_trace(self, frame, w_func): "Profile the call of a builtin function" if self.profilefunc is not None: - self._llprofile('c_call', w_func) + self._trace(frame, 'c_call', w_func) + #self._llprofile('c_call', w_func) - def c_return_llprofile(self, w_retval): + def c_return_trace(self, frame, w_retval): "Profile the return from a builtin function" if self.profilefunc is not None: - self._llprofile('c_return', w_retval) + self._trace(frame, 'c_return', w_retval) + #self._llprofile('c_return', w_retval) - def c_exception_llprofile(self, operationerr): + def c_exception_trace(self, frame, operationerr): "Profile function called upon OperationError." if self.profilefunc is not None: - self._llprofile('c_exception', operationerr) + #self._llprofile('c_exception', operationerr) + self._trace(frame, 'c_exception', operationerr) def _llprofile(self, event, w_arg): fr = self.framestack.items @@ -201,7 +204,7 @@ self.is_tracing = is_tracing def _trace(self, frame, event, w_arg, operr=None): - if self.is_tracing or frame.hide(): + if self.is_tracing or frame.hide() or frame is None: return space = self.space @@ -237,7 +240,7 @@ # Profile cases if self.profilefunc is not None: - if event not in ['leaveframe', 'call']: + if event not in ['leaveframe', 'call', 'c_call', 'c_return']: return last_exception = None Modified: pypy/branch/builtin-profiling/pypy/interpreter/gateway.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/gateway.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/gateway.py Fri Jul 11 14:15:41 2008 @@ -378,6 +378,16 @@ else: return typ.__name__ + '_w' +def getexecutioncontext_and_frame(space): + ctx = space.getexecutioncontext() + if not ctx.framestack.empty(): + frame = ctx.framestack.top() + else: + ctx = None + frame = None + return ctx, frame + + class BuiltinCode(eval.Code): "The code object implementing a built-in (interpreter-level) hook." _immutable_ = True @@ -480,13 +490,15 @@ def funcrun(self, func, args): space = func.space - ctx = space.getexecutioncontext() + #ctx, frame = getexecutioncontext_and_frame(space) activation = self.activation scope_w = args.parse(func.name, self.sig, func.defs_w) try: - ctx.c_call_llprofile(activation._run) + #if frame: + # ctx.c_call_trace(frame, activation._run) w_result = activation._run(space, scope_w) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) @@ -512,11 +524,13 @@ def funcrun(self, func, args): space = func.space - ctx = space.getexecutioncontext() + #ctx, frame = getexecutioncontext_and_frame(space) try: - ctx.c_call_llprofile(self.func__args__) + #if frame: + # ctx.c_call_trace(frame, self.func__args__) w_result = self.func__args__(space, args) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -539,16 +553,18 @@ def funcrun(self, func, args): space = func.space - ctx = space.getexecutioncontext() + #ctx, frame = getexecutioncontext_and_frame(space) try: w_obj, newargs = args.popfirst() except IndexError: return BuiltinCode.funcrun(self, func, args) else: try: - ctx.c_call_llprofile(self.func__args__) + #if frame: + # ctx.c_call_trace(frame, self.func__args__) w_result = self.func__args__(space, w_obj, newargs) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -571,10 +587,12 @@ def fastcall_0(self, space, w_func): self = hint(self, deepfreeze=True) try: - ctx = space.getexecutioncontext() - ctx.c_call_llprofile(self.fastfunc_0) + #ctx, frame = getexecutioncontext_and_frame(space) + #if frame: + # ctx.c_call_trace(frame, self.fastfunc_0) w_result = self.fastfunc_0(space) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -590,10 +608,12 @@ def fastcall_1(self, space, w_func, w1): self = hint(self, deepfreeze=True) try: - ctx = space.getexecutioncontext() - ctx.c_call_llprofile(self.fastfunc_1) + #ctx, frame = getexecutioncontext_and_frame(space) + #if frame: + # ctx.c_call_trace(frame, self.fastfunc_1) w_result = self.fastfunc_1(space, w1) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -616,10 +636,12 @@ def fastcall_2(self, space, w_func, w1, w2): self = hint(self, deepfreeze=True) try: - ctx = space.getexecutioncontext() - ctx.c_call_llprofile(self.fastfunc_2) + #ctx, frame = getexecutioncontext_and_frame(space) + #if frame: + # ctx.c_call_trace(frame, self.fastfunc_2) w_result = self.fastfunc_2(space, w1, w2) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -642,10 +664,12 @@ def fastcall_3(self, space, w_func, w1, w2, w3): self = hint(self, deepfreeze=True) try: - ctx = space.getexecutioncontext() - ctx.c_call_llprofile(self.fastfunc_3) + #ctx, frame = getexecutioncontext_and_frame(space) + #if frame: + # ctx.c_call_trace(frame, self.fastfunc_3) w_result = self.fastfunc_3(space, w1, w2, w3) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: @@ -667,11 +691,13 @@ class BuiltinCode4(BuiltinCode): def fastcall_4(self, space, w_func, w1, w2, w3, w4): self = hint(self, deepfreeze=True) + #ctx, frame = getexecutioncontext_and_frame(space) try: - ctx = space.getexecutioncontext() - ctx.c_call_llprofile(self.fastfunc_4) + #if frame: + #ctx.c_call_trace(frame, self.fastfunc_4) w_result = self.fastfunc_4(space, w1, w2, w3, w4) - ctx.c_return_llprofile(w_result) + #if frame: + # ctx.c_return_trace(frame, w_result) except KeyboardInterrupt: raise OperationError(space.w_KeyboardInterrupt, space.w_None) except MemoryError: Modified: pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Fri Jul 11 14:15:41 2008 @@ -73,9 +73,12 @@ space.appexec([], """(): l = [] l.append(3) + + return l """) space.getexecutioncontext().setllprofile(None, None) from pprint import pprint - pprint([(x, y, z) for x,y,z in l if x in ('call','c_call')]) + pprint(l) + #pprint([(x, y, z) for x,y,z in l if x in ('call','c_call')]) assert l == ['call', 'return', 'call', 'c_call', 'c_return', 'return'] From jlg at codespeak.net Fri Jul 11 14:35:29 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Fri, 11 Jul 2008 14:35:29 +0200 (CEST) Subject: [pypy-svn] r56444 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711123529.48391698120@codespeak.net> Author: jlg Date: Fri Jul 11 14:35:27 2008 New Revision: 56444 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - subclassing itertools not allowed Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 14:35:27 2008 @@ -29,12 +29,7 @@ def W_Count___new__(space, w_subtype, firstval=0): - """ - Create a new count object and call its initializer. - """ - result = space.allocate_instance(W_Count, w_subtype) - W_Count.__init__(result, space, firstval) - return space.wrap(result) + return space.wrap(W_Count(space, firstval)) W_Count.typedef = TypeDef( 'count', @@ -56,6 +51,7 @@ yield n n += 1 """) +W_Count.typedef.acceptable_as_base_class = False class W_Repeat(Wrappable): @@ -82,12 +78,7 @@ return self.space.wrap(self) def W_Repeat___new__(space, w_subtype, w_obj, w_times=None): - """ - Create a new repeat object and call its initializer. - """ - result = space.allocate_instance(W_Repeat, w_subtype) - W_Repeat.__init__(result, space, w_obj, w_times) - return space.wrap(result) + return space.wrap(W_Repeat(space, w_obj, w_times)) W_Repeat.typedef = TypeDef( 'repeat', @@ -110,6 +101,7 @@ for i in xrange(times): yield object """) +W_Repeat.typedef.acceptable_as_base_class = False class W_TakeWhile(Wrappable): @@ -135,9 +127,7 @@ return w_obj def W_TakeWhile___new__(space, w_subtype, w_predicate, w_iterable): - result = space.allocate_instance(W_TakeWhile, w_subtype) - W_TakeWhile.__init__(result, space, w_predicate, w_iterable) - return space.wrap(result) + return space.wrap(W_TakeWhile(space, w_predicate, w_iterable)) W_TakeWhile.typedef = TypeDef( @@ -157,6 +147,7 @@ else: break """) +W_TakeWhile.typedef.acceptable_as_base_class = False class W_DropWhile(Wrappable): @@ -183,9 +174,7 @@ return w_obj def W_DropWhile___new__(space, w_subtype, w_predicate, w_iterable): - result = space.allocate_instance(W_DropWhile, w_subtype) - W_DropWhile.__init__(result, space, w_predicate, w_iterable) - return space.wrap(result) + return space.wrap(W_DropWhile(space, w_predicate, w_iterable)) W_DropWhile.typedef = TypeDef( @@ -209,6 +198,7 @@ for x in iterable: yield x """) +W_DropWhile.typedef.acceptable_as_base_class = False class _IFilterBase(Wrappable): @@ -240,9 +230,7 @@ reverse = False def W_IFilter___new__(space, w_subtype, w_predicate, w_iterable): - result = space.allocate_instance(W_IFilter, w_subtype) - W_IFilter.__init__(result, space, w_predicate, w_iterable) - return space.wrap(result) + return space.wrap(W_IFilter(space, w_predicate, w_iterable)) W_IFilter.typedef = TypeDef( 'ifilter', @@ -262,14 +250,13 @@ if predicate(x): yield x """) +W_IFilter.typedef.acceptable_as_base_class = False class W_IFilterFalse(_IFilterBase): reverse = True def W_IFilterFalse___new__(space, w_subtype, w_predicate, w_iterable): - result = space.allocate_instance(W_IFilterFalse, w_subtype) - W_IFilterFalse.__init__(result, space, w_predicate, w_iterable) - return space.wrap(result) + return space.wrap(W_IFilterFalse(space, w_predicate, w_iterable)) W_IFilterFalse.typedef = TypeDef( 'ifilterfalse', @@ -289,6 +276,7 @@ if not predicate(x): yield x """) +W_IFilterFalse.typedef.acceptable_as_base_class = False class W_ISlice(Wrappable): def __init__(self, space, w_iterable, w_startstop, args_w): @@ -354,9 +342,7 @@ return w_obj def W_ISlice___new__(space, w_subtype, w_iterable, w_startstop, args_w): - result = space.allocate_instance(W_ISlice, w_subtype) - W_ISlice.__init__(result, space, w_iterable, w_startstop, args_w) - return space.wrap(result) + return space.wrap(W_ISlice(space, w_iterable, w_startstop, args_w)) W_ISlice.typedef = TypeDef( 'islice', @@ -375,6 +361,7 @@ internal structure has been flattened (for example, a multi-line report may list a name field on every third line). """) +W_ISlice.typedef.acceptable_as_base_class = False class W_Chain(Wrappable): @@ -426,9 +413,7 @@ return w_obj def W_Chain___new__(space, w_subtype, args_w): - result = space.allocate_instance(W_Chain, w_subtype) - W_Chain.__init__(result, space, args_w) - return space.wrap(result) + return space.wrap(W_Chain(space, args_w)) W_Chain.typedef = TypeDef( 'chain', @@ -447,6 +432,7 @@ for element in it: yield element """) +W_Chain.typedef.acceptable_as_base_class = False class W_IMap(Wrappable): @@ -488,9 +474,7 @@ def W_IMap___new__(space, w_subtype, w_fun, args_w): - result = space.allocate_instance(W_IMap, w_subtype) - W_IMap.__init__(result, space, w_fun, args_w) - return space.wrap(result) + return space.wrap(W_IMap(space, w_fun, args_w)) W_IMap.typedef = TypeDef( 'imap', @@ -518,4 +502,5 @@ yield function(*args) """) +W_IMap.typedef.acceptable_as_base_class = False Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 14:35:27 2008 @@ -341,8 +341,11 @@ (itertools.imap, (None,)), ] for cls, args in iterables: - class A(cls): + try: + class A(cls): + pass + except TypeError: pass - a = A(*args) - assert isinstance(a, A) + else: + assert False, "Subclassing should fail." From adurdin at codespeak.net Fri Jul 11 14:45:53 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 14:45:53 +0200 (CEST) Subject: [pypy-svn] r56445 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711124553.23E9E398005@codespeak.net> Author: adurdin Date: Fri Jul 11 14:45:51 2008 New Revision: 56445 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - izip() implemented. Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Fri Jul 11 14:45:51 2008 @@ -32,6 +32,7 @@ 'ifilterfalse' : 'interp_itertools.W_IFilterFalse', 'imap' : 'interp_itertools.W_IMap', 'islice' : 'interp_itertools.W_ISlice', + 'izip' : 'interp_itertools.W_IZip', 'repeat' : 'interp_itertools.W_Repeat', 'takewhile' : 'interp_itertools.W_TakeWhile', } Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 14:45:51 2008 @@ -504,3 +504,33 @@ """) W_IMap.typedef.acceptable_as_base_class = False + +class W_IZip(W_IMap): + + def __init__(self, space, args_w): + super(W_IZip, self).__init__(space, space.w_None, args_w) + + +def W_IZip___new__(space, w_subtype, args_w): + return space.wrap(W_IZip(space, args_w)) + +W_IZip.typedef = TypeDef( + 'izip', + __new__ = interp2app(W_IZip___new__, unwrap_spec=[ObjSpace, W_Root, 'args_w']), + __iter__ = interp2app(W_IZip.iter_w, unwrap_spec=['self']), + next = interp2app(W_IZip.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that aggregates elements from each of the + iterables. Like zip() except that it returns an iterator instead + of a list. Used for lock-step iteration over several iterables at + a time. + + Equivalent to : + + def izip(*iterables): + iterables = map(iter, iterables) + while iterables: + result = [i.next() for i in iterables] + yield tuple(result) + """) +W_IZip.typedef.acceptable_as_base_class = False + Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 14:45:51 2008 @@ -17,6 +17,7 @@ itertools.islice([], 0), itertools.chain(), itertools.imap(None), + itertools.izip(), ] for it in iterables: @@ -308,6 +309,26 @@ assert it.next() == x raises(StopIteration, it.next) + def test_izip(self): + import itertools + + it = itertools.izip() + raises(StopIteration, it.next) + + obj_list = [object(), object(), object()] + it = itertools.izip(obj_list) + for x in obj_list: + assert it.next() == (x, ) + raises(StopIteration, it.next) + + it = itertools.izip([1, 2, 3], [4], [5, 6]) + assert it.next() == (1, 4, 5) + raises(StopIteration, it.next) + + it = itertools.izip([], [], [1], []) + raises(StopIteration, it.next) + + def test_docstrings(self): import itertools @@ -322,6 +343,7 @@ itertools.islice, itertools.chain, itertools.imap, + itertools.izip, ] for method in methods: assert method.__doc__ @@ -339,6 +361,7 @@ (itertools.islice, ([], 0)), (itertools.chain, ()), (itertools.imap, (None,)), + (itertools.izip, ()), ] for cls, args in iterables: try: From adurdin at codespeak.net Fri Jul 11 14:54:37 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 14:54:37 +0200 (CEST) Subject: [pypy-svn] r56446 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711125437.2C37639B596@codespeak.net> Author: adurdin Date: Fri Jul 11 14:54:36 2008 New Revision: 56446 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools.izip behaves per spec after exhausting. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 14:54:36 2008 @@ -465,7 +465,12 @@ if not self.iterators_w: raise OperationError(self.space.w_StopIteration, self.space.w_None) - w_objects = [self.space.next(w_it) for w_it in self.iterators_w] + try: + w_objects = [self.space.next(w_it) for w_it in self.iterators_w] + except OperationError, e: + if e.match(self.space, self.space.w_StopIteration): + self.iterators_w = None + raise if self.identity_fun: return self.space.newtuple(w_objects) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 14:54:36 2008 @@ -327,7 +327,17 @@ it = itertools.izip([], [], [1], []) raises(StopIteration, it.next) - + + # Up to one additional item may be consumed per iterable, as per python docs + it1 = iter([1, 2, 3, 4, 5, 6]) + it2 = iter([5, 6]) + it = itertools.izip(it1, it2) + for x in [(1, 5), (2, 6)]: + assert it.next() == x + raises(StopIteration, it.next) + assert it1.next() == 4 + raises(StopIteration, it.next) + assert it1.next() == 5 def test_docstrings(self): import itertools From arigo at codespeak.net Fri Jul 11 15:10:10 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 11 Jul 2008 15:10:10 +0200 (CEST) Subject: [pypy-svn] r56447 - pypy/dist/pypy/interpreter Message-ID: <20080711131010.3E64A39B592@codespeak.net> Author: arigo Date: Fri Jul 11 15:10:09 2008 New Revision: 56447 Modified: pypy/dist/pypy/interpreter/baseobjspace.py pypy/dist/pypy/interpreter/pyframe.py Log: Performance improvement: hide the base PyFrame class from the annotator; only see the StdObjSpaceFrame subclass. Modified: pypy/dist/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/dist/pypy/interpreter/baseobjspace.py (original) +++ pypy/dist/pypy/interpreter/baseobjspace.py Fri Jul 11 15:10:09 2008 @@ -229,6 +229,9 @@ self.actionflag.register_action(self.frame_trace_action) self.setoptions(**kw) + from pypy.interpreter.pyframe import PyFrame + self.FrameClass = PyFrame # can be overridden to a subclass + # if self.config.objspace.logbytecodes: # self.bytecodecounts = {} @@ -484,8 +487,7 @@ def createframe(self, code, w_globals, closure=None): "Create an empty PyFrame suitable for this code object." - from pypy.interpreter import pyframe - return pyframe.PyFrame(self, code, w_globals, closure) + return self.FrameClass(self, code, w_globals, closure) def allocate_lock(self): """Return an interp-level Lock object if threads are enabled, Modified: pypy/dist/pypy/interpreter/pyframe.py ============================================================================== --- pypy/dist/pypy/interpreter/pyframe.py (original) +++ pypy/dist/pypy/interpreter/pyframe.py Fri Jul 11 15:10:09 2008 @@ -101,6 +101,10 @@ def execute_frame(self): """Execute this frame. Main entry point to the interpreter.""" from pypy.rlib import rstack + # the following 'assert' is an annotation hint: it hides from + # the annotator all methods that are defined in PyFrame but + # overridden in the FrameClass subclass of PyFrame. + assert isinstance(self, self.space.FrameClass) executioncontext = self.space.getexecutioncontext() executioncontext.enter(self) try: From hpk at codespeak.net Fri Jul 11 15:16:13 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Fri, 11 Jul 2008 15:16:13 +0200 (CEST) Subject: [pypy-svn] r56448 - in pypy/dist: lib-python pypy/lib Message-ID: <20080711131613.8937139B592@codespeak.net> Author: hpk Date: Fri Jul 11 15:16:12 2008 New Revision: 56448 Added: pypy/dist/pypy/lib/hashlib.py (contents, props changed) Modified: pypy/dist/lib-python/conftest.py Log: add a stripped down hashlib module that supports md5 and sha1, with stripped down tests. Modified: pypy/dist/lib-python/conftest.py ============================================================================== --- pypy/dist/lib-python/conftest.py (original) +++ pypy/dist/lib-python/conftest.py Fri Jul 11 15:16:12 2008 @@ -523,6 +523,9 @@ RegrTest('test_gzip.py', enabled=False, dumbtest=1), RegrTest('test_hash.py', enabled=True, core=True), + RegrTest('test_hashlib.py', enabled=True, core=True), + # test_hashlib is stripped down, originates from 2.5 + RegrTest('test_heapq.py', enabled=True, core=True), RegrTest('test_hexoct.py', enabled=True, core=True), RegrTest('test_hmac.py', enabled=True), Added: pypy/dist/pypy/lib/hashlib.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/hashlib.py Fri Jul 11 15:16:12 2008 @@ -0,0 +1,127 @@ +# XXX ported version of the original hashlib.py from python +# XXX 2.5, only supports md5 and sha1 by using the existing +# XXX modules of PyPy +# $Id: hashlib.py 52533 2006-10-29 18:01:12Z georg.brandl $ +# +# Copyright (C) 2005 Gregory P. Smith (greg at electricrain.com) +# Licensed to PSF under a Contributor Agreement. +# + +__doc__ = """hashlib module - A common interface to many hash functions. + +new(name, string='') - returns a new hash object implementing the + given hash function; initializing the hash + using the given string data. + +Named constructor functions are also available, these are much faster +than using new(): + +md5(), sha1(), sha224(), sha256(), sha384(), and sha512() + +More algorithms may be available on your platform but the above are +guaranteed to exist. + +Choose your hash function wisely. Some have known collision weaknesses. +sha384 and sha512 will be slow on 32 bit platforms. + +Hash objects have these methods: + - update(arg): Update the hash object with the string arg. Repeated calls + are equivalent to a single call with the concatenation of all + the arguments. + - digest(): Return the digest of the strings passed to the update() method + so far. This may contain non-ASCII characters, including + NUL bytes. + - hexdigest(): Like digest() except the digest is returned as a string of + double length, containing only hexadecimal digits. + - copy(): Return a copy (clone) of the hash object. This can be used to + efficiently compute the digests of strings that share a common + initial substring. + +For example, to obtain the digest of the string 'Nobody inspects the +spammish repetition': + + >>> import hashlib + >>> m = hashlib.md5() + >>> m.update("Nobody inspects") + >>> m.update(" the spammish repetition") + >>> m.digest() + '\xbbd\x9c\x83\xdd\x1e\xa5\xc9\xd9\xde\xc9\xa1\x8d\xf0\xff\xe9' + +More condensed: + + >>> hashlib.sha224("Nobody inspects the spammish repetition").hexdigest() + 'a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2' + +""" + + +def __get_builtin_constructor(name): + if name in ('SHA1', 'sha1'): + import sha + return sha.new + elif name in ('MD5', 'md5'): + import md5 + return md5.new + # XXX code for other codecs deleted here + raise ValueError, "unsupported hash type" + + +def __py_new(name, string=''): + """new(name, string='') - Return a new hashing object using the named algorithm; + optionally initialized with a string. + """ + return __get_builtin_constructor(name)(string) + + +def __hash_new(name, string=''): + """new(name, string='') - Return a new hashing object using the named algorithm; + optionally initialized with a string. + """ + try: + return _hashlib.new(name, string) + except ValueError: + # If the _hashlib module (OpenSSL) doesn't support the named + # hash, try using our builtin implementations. + # This allows for SHA224/256 and SHA384/512 support even though + # the OpenSSL library prior to 0.9.8 doesn't provide them. + return __get_builtin_constructor(name)(string) + + +try: + import _hashlib + # use the wrapper of the C implementation + new = __hash_new + + for opensslFuncName in filter(lambda n: n.startswith('openssl_'), dir(_hashlib)): + funcName = opensslFuncName[len('openssl_'):] + try: + # try them all, some may not work due to the OpenSSL + # version not supporting that algorithm. + f = getattr(_hashlib, opensslFuncName) + f() + # Use the C function directly (very fast) + exec funcName + ' = f' + except ValueError: + try: + # Use the builtin implementation directly (fast) + exec funcName + ' = __get_builtin_constructor(funcName)' + except ValueError: + # this one has no builtin implementation, don't define it + pass + # clean up our locals + del f + del opensslFuncName + del funcName + +except ImportError: + # We don't have the _hashlib OpenSSL module? + # use the built in legacy interfaces via a wrapper function + new = __py_new + + # lookup the C function to use directly for the named constructors + md5 = __get_builtin_constructor('md5') + sha1 = __get_builtin_constructor('sha1') + #sha224 = __get_builtin_constructor('sha224') + #sha256 = __get_builtin_constructor('sha256') + #sha384 = __get_builtin_constructor('sha384') + #sha512 = __get_builtin_constructor('sha512') From adurdin at codespeak.net Fri Jul 11 15:21:36 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 15:21:36 +0200 (CEST) Subject: [pypy-svn] r56449 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711132136.A91F639B592@codespeak.net> Author: adurdin Date: Fri Jul 11 15:21:34 2008 New Revision: 56449 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) Complying to CPython docs for behaviour with invalid arguments. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 15:21:34 2008 @@ -435,6 +435,7 @@ W_Chain.typedef.acceptable_as_base_class = False class W_IMap(Wrappable): + _error_name = "imap" def __init__(self, space, w_fun, args_w): self.space = space @@ -448,7 +449,7 @@ iterator_w = space.iter(iterable_w) except OperationError, e: if e.match(self.space, self.space.w_TypeError): - raise OperationError(space.w_TypeError, space.wrap("imap argument #" + str(i + 1) + " must support iteration")) + raise OperationError(space.w_TypeError, space.wrap(self._error_name + " argument #" + str(i + 1) + " must support iteration")) else: raise else: @@ -511,6 +512,7 @@ class W_IZip(W_IMap): + _error_name = "izip" def __init__(self, space, args_w): super(W_IZip, self).__init__(space, space.w_None, args_w) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 15:21:34 2008 @@ -278,6 +278,8 @@ itertools.chain(*args) except TypeError, e: assert str(e) == "chain argument #%d must support iteration" % (x + 1) + else: + fail("TypeError expected") def test_imap(self): import itertools @@ -309,6 +311,18 @@ assert it.next() == x raises(StopIteration, it.next) + def test_imap_wrongargs(self): + import itertools + + # Duplicate python 2.4 behaviour for invalid arguments + it = itertools.imap(0) + raises(StopIteration, it.next) + it = itertools.imap(0, []) + raises(StopIteration, it.next) + it = itertools.imap(0, [0]) + raises(TypeError, it.next) + raises(TypeError, itertools.imap, None, 0) + def test_izip(self): import itertools @@ -339,6 +353,21 @@ raises(StopIteration, it.next) assert it1.next() == 5 + def test_izip_wrongargs(self): + import itertools + + # Duplicate python 2.4 behaviour for invalid arguments + raises(TypeError, itertools.izip, None, 0) + + for x in range(10): + args = [()] * x + [None] + [()] * (9 - x) + try: + itertools.izip(*args) + except TypeError, e: + assert str(e) == "izip argument #%d must support iteration" % (x + 1) + else: + fail("TypeError expected") + def test_docstrings(self): import itertools From adurdin at codespeak.net Fri Jul 11 15:40:17 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 15:40:17 +0200 (CEST) Subject: [pypy-svn] r56450 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711134017.DEDA3169E3A@codespeak.net> Author: adurdin Date: Fri Jul 11 15:40:15 2008 New Revision: 56450 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - added cycle() Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Fri Jul 11 15:40:15 2008 @@ -25,16 +25,17 @@ """ interpleveldefs = { - 'chain' : 'interp_itertools.W_Chain', - 'count' : 'interp_itertools.W_Count', - 'dropwhile' : 'interp_itertools.W_DropWhile', - 'ifilter' : 'interp_itertools.W_IFilter', - 'ifilterfalse' : 'interp_itertools.W_IFilterFalse', - 'imap' : 'interp_itertools.W_IMap', - 'islice' : 'interp_itertools.W_ISlice', - 'izip' : 'interp_itertools.W_IZip', - 'repeat' : 'interp_itertools.W_Repeat', - 'takewhile' : 'interp_itertools.W_TakeWhile', + 'chain' : 'interp_itertools.W_Chain', + 'count' : 'interp_itertools.W_Count', + 'cycle' : 'interp_itertools.W_Cycle', + 'dropwhile' : 'interp_itertools.W_DropWhile', + 'ifilter' : 'interp_itertools.W_IFilter', + 'ifilterfalse' : 'interp_itertools.W_IFilterFalse', + 'imap' : 'interp_itertools.W_IMap', + 'islice' : 'interp_itertools.W_ISlice', + 'izip' : 'interp_itertools.W_IZip', + 'repeat' : 'interp_itertools.W_Repeat', + 'takewhile' : 'interp_itertools.W_TakeWhile', } appleveldefs = { Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 15:40:15 2008 @@ -541,3 +541,46 @@ """) W_IZip.typedef.acceptable_as_base_class = False + +class W_Cycle(Wrappable): + + def __init__(self, space, w_iterable): + self.space = space + self.saved_w = [] + self.w_iterable = space.iter(w_iterable) + self.saved_iterator = None + self.exhausted = False + + def next_w(self): + if self.exhausted: + if not self.saved_w: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + try: + w_obj = self.saved_iterator.next() + except StopIteration: + self.saved_iterator = iter(self.saved_w) + w_obj = self.saved_iterator.next() + else: + try: + w_obj = self.space.next(self.w_iterable) + except OperationError, e: + if e.match(self.space, self.space.w_StopIteration): + self.exhausted = True + if not self.saved_w: + raise + self.saved_iterator = iter(self.saved_w) + w_obj = self.saved_iterator.next() + else: + raise + else: + self.saved_w.append(w_obj) + return w_obj + +def W_Cycle___new__(space, w_subtype, w_iterable): + return space.wrap(W_Cycle(space, w_iterable)) + +W_Cycle.typedef = TypeDef( + 'cycle', + __new__ = interp2app(W_Cycle___new__, unwrap_spec=[ObjSpace, W_Root, W_Root]), + next = interp2app(W_Cycle.next_w, unwrap_spec=['self'])) +W_Cycle.typedef.acceptable_as_base_class = False Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 15:40:15 2008 @@ -368,6 +368,17 @@ else: fail("TypeError expected") + def test_cycle(self): + import itertools + + it = itertools.cycle([]) + raises(StopIteration, it.next) + + it = itertools.cycle([1, 2, 3]) + for x in [1, 2, 3, 1, 2, 3, 1, 2, 3]: + assert it.next() == x + + def test_docstrings(self): import itertools From adurdin at codespeak.net Fri Jul 11 16:11:58 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 16:11:58 +0200 (CEST) Subject: [pypy-svn] r56451 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711141158.2095616A0C9@codespeak.net> Author: adurdin Date: Fri Jul 11 16:11:57 2008 New Revision: 56451 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) tidying tests for interp_itertools Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 16:11:57 2008 @@ -551,6 +551,9 @@ self.saved_iterator = None self.exhausted = False + def iter_w(self): + return self.space.wrap(self) + def next_w(self): if self.exhausted: if not self.saved_w: @@ -582,5 +585,21 @@ W_Cycle.typedef = TypeDef( 'cycle', __new__ = interp2app(W_Cycle___new__, unwrap_spec=[ObjSpace, W_Root, W_Root]), - next = interp2app(W_Cycle.next_w, unwrap_spec=['self'])) + __iter__ = interp2app(W_Cycle.iter_w, unwrap_spec=['self']), + next = interp2app(W_Cycle.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator returning elements from the iterable and + saving a copy of each. When the iterable is exhausted, return + elements from the saved copy. Repeats indefinitely. + + Equivalent to : + + def cycle(iterable): + saved = [] + for element in iterable: + yield element + saved.append(element) + while saved: + for element in saved: + yield element + """) W_Cycle.typedef.acceptable_as_base_class = False Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 16:11:57 2008 @@ -4,28 +4,6 @@ def setup_class(cls): cls.space = gettestobjspace(usemodules=['itertools']) - def test_iterables(self): - import itertools - - iterables = [ - itertools.count(), - itertools.repeat(None), - itertools.takewhile(bool, []), - itertools.dropwhile(bool, []), - itertools.ifilter(None, []), - itertools.ifilterfalse(None, []), - itertools.islice([], 0), - itertools.chain(), - itertools.imap(None), - itertools.izip(), - ] - - for it in iterables: - assert hasattr(it, '__iter__') - assert iter(it) is it - assert hasattr(it, 'next') - assert callable(it.next) - def test_count(self): import itertools @@ -378,40 +356,65 @@ for x in [1, 2, 3, 1, 2, 3, 1, 2, 3]: assert it.next() == x - + + def test_iterables(self): + import itertools + + iterables = [ + itertools.chain(), + itertools.count(), + itertools.cycle([]), + itertools.dropwhile(bool, []), + itertools.ifilter(None, []), + itertools.ifilterfalse(None, []), + itertools.imap(None), + itertools.islice([], 0), + itertools.izip(), + itertools.repeat(None), + itertools.takewhile(bool, []), + ] + + for it in iterables: + assert hasattr(it, '__iter__') + assert iter(it) is it + assert hasattr(it, 'next') + assert callable(it.next) + def test_docstrings(self): import itertools assert itertools.__doc__ methods = [ + itertools.chain, itertools.count, - itertools.repeat, - itertools.takewhile, + itertools.cycle, itertools.dropwhile, itertools.ifilter, itertools.ifilterfalse, - itertools.islice, - itertools.chain, itertools.imap, + itertools.islice, itertools.izip, + itertools.repeat, + itertools.takewhile, ] for method in methods: assert method.__doc__ - + def test_subclassing(self): import itertools - # not sure how useful this actually is, but CPython implements it + # Although implemented as classes, the itertools functions should not be subclassable iterables = [ + (itertools.chain, ()), (itertools.count, ()), - (itertools.repeat, (None,)), - (itertools.takewhile, (bool, [])), + (itertools.cycle, ()), (itertools.dropwhile, (bool, [])), (itertools.ifilter, (None, [])), (itertools.ifilterfalse, (None, [])), - (itertools.islice, ([], 0)), - (itertools.chain, ()), (itertools.imap, (None,)), + (itertools.islice, ([], 0)), (itertools.izip, ()), + (itertools.repeat, (None,)), + (itertools.takewhile, (bool, [])), ] for cls, args in iterables: try: @@ -421,4 +424,4 @@ pass else: assert False, "Subclassing should fail." - + From jlg at codespeak.net Fri Jul 11 16:26:01 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Fri, 11 Jul 2008 16:26:01 +0200 (CEST) Subject: [pypy-svn] r56452 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711142601.48C4239B59D@codespeak.net> Author: jlg Date: Fri Jul 11 16:25:59 2008 New Revision: 56452 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - starmap implemented Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Fri Jul 11 16:25:59 2008 @@ -35,6 +35,7 @@ 'islice' : 'interp_itertools.W_ISlice', 'izip' : 'interp_itertools.W_IZip', 'repeat' : 'interp_itertools.W_Repeat', + 'starmap' : 'interp_itertools.W_StarMap', 'takewhile' : 'interp_itertools.W_TakeWhile', } Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 16:25:59 2008 @@ -514,12 +514,8 @@ class W_IZip(W_IMap): _error_name = "izip" - def __init__(self, space, args_w): - super(W_IZip, self).__init__(space, space.w_None, args_w) - - def W_IZip___new__(space, w_subtype, args_w): - return space.wrap(W_IZip(space, args_w)) + return space.wrap(W_IZip(space, space.w_None, args_w)) W_IZip.typedef = TypeDef( 'izip', @@ -603,3 +599,46 @@ yield element """) W_Cycle.typedef.acceptable_as_base_class = False + +class W_StarMap(Wrappable): + + def __init__(self, space, w_fun, w_iterable): + self.space = space + self.w_fun = w_fun + self.w_iterable = self.space.iter(w_iterable) + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + w_obj = self.space.next(self.w_iterable) + if not self.space.is_true(self.space.isinstance(w_obj, self.space.w_tuple)): + raise OperationError(self.space.w_TypeError, self.space.wrap("iterator must return a tuple")) + + return self.space.call(self.w_fun, w_obj) + +def W_StarMap___new__(space, w_subtype, w_fun,w_iterable): + return space.wrap(W_StarMap(space, w_fun, w_iterable)) + +W_StarMap.typedef = TypeDef( + 'starmap', + __new__ = interp2app(W_StarMap___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), + __iter__ = interp2app(W_StarMap.iter_w, unwrap_spec=['self']), + next = interp2app(W_StarMap.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that computes the function using arguments + tuples obtained from the iterable. Used instead of imap() when + argument parameters are already grouped in tuples from a single + iterable (the data has been ``pre-zipped''). The difference + between imap() and starmap() parallels the distinction between + function(a,b) and function(*c). + + Equivalent to : + + def starmap(function, iterable): + iterable = iter(iterable) + while True: + yield function(*iterable.next()) + """) +W_StarMap.typedef.acceptable_as_base_class = False + + Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 16:25:59 2008 @@ -356,6 +356,29 @@ for x in [1, 2, 3, 1, 2, 3, 1, 2, 3]: assert it.next() == x + def test_starmap(self): + import itertools, operator + + it = itertools.starmap(operator.add, []) + raises(StopIteration, it.next) + + it = itertools.starmap(operator.add, [(0, 1), (2, 3), (4, 5)]) + for x in [1, 5, 9]: + assert it.next() == x + raises(StopIteration, it.next) + + def test_starmap_wrongargs(self): + import itertools + + it = itertools.starmap(None, [(1, )]) + raises(TypeError, it.next) + + it = itertools.starmap(None, []) + raises(StopIteration, it.next) + + it = itertools.starmap(bool, [0]) + raises(TypeError, it.next) + def test_iterables(self): import itertools @@ -371,6 +394,7 @@ itertools.islice([], 0), itertools.izip(), itertools.repeat(None), + itertools.starmap(bool, []), itertools.takewhile(bool, []), ] @@ -395,6 +419,7 @@ itertools.islice, itertools.izip, itertools.repeat, + itertools.starmap, itertools.takewhile, ] for method in methods: @@ -414,6 +439,7 @@ (itertools.islice, ([], 0)), (itertools.izip, ()), (itertools.repeat, (None,)), + (itertools.starmap, (bool, [])), (itertools.takewhile, (bool, [])), ] for cls, args in iterables: From antocuni at codespeak.net Fri Jul 11 16:26:19 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 11 Jul 2008 16:26:19 +0200 (CEST) Subject: [pypy-svn] r56453 - in pypy/branch/builtin-profiling/pypy/interpreter: . test Message-ID: <20080711142619.CE93139B59A@codespeak.net> Author: antocuni Date: Fri Jul 11 16:26:19 2008 New Revision: 56453 Modified: pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py pypy/branch/builtin-profiling/pypy/interpreter/function.py pypy/branch/builtin-profiling/pypy/interpreter/pyframe.py pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Log: (antocuni, arigo, stephan) (in-progress) trace calls/returns/exceptions of builtin functions. It still doesn't cover all the possible cases Modified: pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py Fri Jul 11 16:26:19 2008 @@ -694,9 +694,42 @@ return self.call_args(w_func, args) def call_valuestack(self, w_func, nargs, frame): + from pypy.interpreter.function import Function, Method + from pypy.interpreter.gateway import BuiltinCode + if frame.is_being_profiled: + ec = self.getexecutioncontext() + is_c_call = False + if isinstance(w_func, Method): + code = w_func.w_function.getcode() + elif isinstance(w_func, Function): + code = w_func.getcode() + if isinstance(code, BuiltinCode): + is_c_call = True + + # XXX: this code is copied&pasted :-( from the slow path + # below. The profiling info could be not very accurate + # because by doing this we disable fast paths when calling + # the function + args = frame.make_arguments(nargs) + try: + try: + if is_c_call: + ec.c_call_trace(frame, w_func) + w_res = self.call_args(w_func, args) + if is_c_call: + ec.c_return_trace(frame, w_res) + return w_res + except OperationError, e: + if is_c_call: + ec.c_exception_trace(frame, e) + raise + finally: + if isinstance(args, ArgumentsFromValuestack): + args.frame = None + + if not self.config.objspace.disable_call_speedhacks: # XXX start of hack for performance - from pypy.interpreter.function import Function, Method hint(w_func.__class__, promote=True) if isinstance(w_func, Method): w_inst = w_func.w_instance Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py Fri Jul 11 16:26:19 2008 @@ -136,6 +136,8 @@ "Trace the call of a function" if self.w_tracefunc is not None or self.profilefunc is not None: self._trace(frame, 'call', self.space.w_None) + if self.profilefunc: + frame.is_being_profiled = True def return_trace(self, frame, w_retval): "Trace the return from a function" @@ -240,7 +242,7 @@ # Profile cases if self.profilefunc is not None: - if event not in ['leaveframe', 'call', 'c_call', 'c_return']: + if event not in ['leaveframe', 'call', 'c_call', 'c_return', 'c_exception']: return last_exception = None Modified: pypy/branch/builtin-profiling/pypy/interpreter/function.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/function.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/function.py Fri Jul 11 16:26:19 2008 @@ -30,7 +30,7 @@ def __repr__(self): # return "function %s.%s" % (self.space, self.name) # maybe we want this shorter: - return "" % self.name + return "<%s %s>" % (self.__class__.__name__, self.name) def call_args(self, args): return self.code.funcrun(self, args) # delegate activation to code Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyframe.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/pyframe.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/pyframe.py Fri Jul 11 16:26:19 2008 @@ -44,6 +44,7 @@ instr_lb = 0 instr_ub = -1 instr_prev = -1 + is_being_profiled = False def __init__(self, space, code, w_globals, closure): self = hint(self, access_directly=True) Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Fri Jul 11 16:26:19 2008 @@ -872,6 +872,8 @@ def CALL_FUNCTION(f, oparg, *ignored): from pypy.rlib import rstack # for resume points + import pdb;pdb.set_trace() + # XXX start of hack for performance if (oparg >> 8) & 0xff == 0: # Only positional arguments Modified: pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Fri Jul 11 16:26:19 2008 @@ -66,19 +66,60 @@ def profile_func(space, w_arg, frame, event, w_aarg): assert w_arg is space.w_None - l.append((event, frame, w_aarg)) + l.append(event) space = self.space space.getexecutioncontext().setllprofile(profile_func, space.w_None) space.appexec([], """(): - l = [] - l.append(3) - - return l + pass """) space.getexecutioncontext().setllprofile(None, None) - from pprint import pprint - pprint(l) - #pprint([(x, y, z) for x,y,z in l if x in ('call','c_call')]) - assert l == ['call', 'return', 'call', 'c_call', 'c_return', 'return'] + assert l == ['call', 'return', 'call', 'return'] + + def test_llprofile_c_call(self): + l = [] + + def profile_func(space, w_arg, frame, event, w_aarg): + assert w_arg is space.w_None + l.append(event) + + space = self.space + space.getexecutioncontext().setllprofile(profile_func, space.w_None) + + def check_snippet(snippet): + space.appexec([], """(): + %s + return + """ % snippet) + space.getexecutioncontext().setllprofile(None, None) + assert l == ['call', 'return', 'call', 'c_call', 'c_return', 'return'] + + check_snippet('l = []; l.append(42)') + check_snippet('max(1, 2)') + check_snippet('args = (1, 2); max(*args)') + check_snippet('max(1, 2, **{})') + check_snippet('args = (1, 2); max(*args, **{})') + + def test_llprofile_c_exception(self): + l = [] + + def profile_func(space, w_arg, frame, event, w_aarg): + assert w_arg is space.w_None + l.append(event) + + space = self.space + space.getexecutioncontext().setllprofile(profile_func, space.w_None) + + def check_snippet(snippet): + space.appexec([], """(): + try: + %s + except: + pass + return + """ % snippet) + space.getexecutioncontext().setllprofile(None, None) + assert l == ['call', 'return', 'call', 'c_call', 'c_exception', 'return'] + + check_snippet('d = {}; d.__getitem__(42)') From antocuni at codespeak.net Fri Jul 11 16:26:57 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 11 Jul 2008 16:26:57 +0200 (CEST) Subject: [pypy-svn] r56454 - pypy/branch/builtin-profiling/pypy/interpreter Message-ID: <20080711142657.C3F5139B59A@codespeak.net> Author: antocuni Date: Fri Jul 11 16:26:57 2008 New Revision: 56454 Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Log: remove pdb Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Fri Jul 11 16:26:57 2008 @@ -872,8 +872,6 @@ def CALL_FUNCTION(f, oparg, *ignored): from pypy.rlib import rstack # for resume points - import pdb;pdb.set_trace() - # XXX start of hack for performance if (oparg >> 8) & 0xff == 0: # Only positional arguments From hruske at codespeak.net Fri Jul 11 16:50:06 2008 From: hruske at codespeak.net (hruske at codespeak.net) Date: Fri, 11 Jul 2008 16:50:06 +0200 (CEST) Subject: [pypy-svn] r56455 - in pypy/dist: lib-python/modified-2.4.1/test pypy/lib Message-ID: <20080711145006.E002039B59E@codespeak.net> Author: hruske Date: Fri Jul 11 16:50:05 2008 New Revision: 56455 Added: pypy/dist/lib-python/modified-2.4.1/test/test_dbm.py (contents, props changed) - copied, changed from r56451, pypy/dist/lib-python/2.4.1/test/test_dbm.py pypy/dist/pypy/lib/dbm.py Log: Adding my ctypes implementation of dbm module, also slightly extended the tests. Added: pypy/dist/pypy/lib/dbm.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/dbm.py Fri Jul 11 16:50:05 2008 @@ -0,0 +1,195 @@ +from ctypes import * +import ctypes.util +import os + +_singleton = 'one' + +class error(Exception): + def __init__(self, msg): + self.msg = msg + + def __str__(self): + return self.msg + +class datum(Structure): + _fields_ = [ + ('dptr', c_char_p), + ('dsize', c_int), + ] + +class dbm(object): + def __init__(self, dbmobj): + self._aobj = dbmobj + + def close(self): + getattr(lib, funcs['close'])(self._aobj) + self._aobj = None + + def keys(self): + if not self._aobj: + raise error('DBM object has already been closed') + allkeys = [] + k = getattr(lib, funcs['firstkey'])(self._aobj) + while k.dptr: + allkeys.append(k.dptr[:k.dsize]) + k = getattr(lib, funcs['nextkey'])(self._aobj) + return allkeys + + def get(self, key, default=_singleton): + if not self._aobj: + raise error('DBM object has already been closed') + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + k = getattr(lib, funcs['fetch'])(self._aobj, dat) + if k.dptr: + return k.dptr[:k.dsize] + if default is _singleton: + raise KeyError + if getattr(lib, funcs['error'])(self._aobj): + getattr(lib, funcs['clearerr'])(self._aobj) + raise error("") + return default + + def __len__(self): + return len(self.keys()) + + def __getitem__(self, key): + assert isinstance(key, str) + value = self.get(key) + if value is None: + raise KeyError + + def _set(self, key, value): + if not self._aobj: + raise error('DBM object has already been closed') + if not isinstance(key, str): + raise TypeError("dbm mappings have string indices only") + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + if value == None: + status = getattr(lib, funcs['delete'])(self._aobj, dat) + if status < 0: + getattr(lib, funcs['clearerr'])(self._aobj) + raise KeyError(key) + else: + if not isinstance(value, str): + raise TypeError("dbm mappings have string indices only") + data = datum() + data.dptr = c_char_p(value) + data.dsize = c_int(len(value)) + status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_INSERT) + if status == 1: + status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_REPLACE) + if getattr(lib, funcs['error'])(self._aobj): + getattr(lib, funcs['clearerr'])(self._aobj) + raise error("") + return status + + def setdefault(self, key, default=None): + if not self._aobj: + raise error('DBM object has already been closed') + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + k = getattr(lib, funcs['fetch'])(self._aobj, dat) + if k.dptr: + return k.dptr[:k.dsize] + if default: + data = datum() + data.dptr = c_char_p(default) + data.dsize = c_int(len(default)) + status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_INSERT) + if status < 0: + getattr(lib, funcs['clearerr'])(self._aobj) + raise error("cannot add item to database") + return default + return None + + def has_key(self, key): + if not self._aobj: + raise error('DBM object has already been closed') + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + k = getattr(lib, funcs['fetch'])(self._aobj, dat) + if k.dptr: + return True + return False + + def __setitem__(self, key, value): + if not isinstance(key, str) and isinstance(value, str): + raise error("dbm mappings have string indices only") + self._set(key, value) + +# initialization for Berkeley DB +_bdb_funcs = { + 'open': '__db_ndbm_open', + 'close': '__db_ndbm_close', + 'firstkey': '__db_ndbm_firstkey', + 'nextkey': '__db_ndbm_nextkey', + 'fetch': '__db_ndbm_fetch', + 'store': '__db_ndbm_store', + 'error': '__db_ndbm_error', + 'delete': '__db_ndbm_delete', +} + +_normal_funcs = { + 'open': 'dbm_open', + 'close': 'dbm_close', + 'firstkey': 'dbm_firstkey', + 'nextkey': 'dbm_nextkey', + 'fetch': 'dbm_fetch', + 'store': 'dbm_store', + 'error': 'dbm_error', + 'delete': 'dbm_delete', +} + +try: + libpath = ctypes.util.find_library('db') + if not libpath: raise + lib = CDLL(libpath) # Linux + _platform = 'bdb' + lib.__db_ndbm_open.argtypes = [c_char_p, c_int, c_int] + lib.__db_ndbm_close.restype = c_void_p + lib.__db_ndbm_firstkey.restype = datum + lib.__db_ndbm_nextkey.restype = datum + lib.__db_ndbm_fetch.restype = datum + lib.__db_ndbm_store.restype = c_int + funcs = _bdb_funcs +except: + lib = CDLL("/usr/lib/libdbm.dylib") # OS X + _platform = 'osx' + lib.dbm_open.argtypes = [c_char_p, c_int, c_int] + lib.dbm_close.restype = c_void_p + lib.dbm_firstkey.restype = datum + lib.dbm_nextkey.restype = datum + lib.dbm_fetch.restype = datum + lib.dbm_store.restype = c_int + funcs = _normal_funcs + + + +lib.DBM_INSERT = 0 +lib.DBM_REPLACE = 1 + +def open(filename, flag='r', mode=0666): + "open a DBM database" + openflag = 0 + + try: + openflag = { + 'r': os.O_RDONLY, + 'rw': os.O_RDWR, + 'w': os.O_RDWR | os.O_CREAT, + 'c': os.O_RDWR | os.O_CREAT, + 'n': os.O_RDWR | os.O_CREAT | os.O_TRUNC, + }[flag] + except KeyError, e: + raise error("arg 2 to open should be 'r', 'w', 'c', or 'n'") + + a_db = getattr(lib, funcs['open'])(filename, openflag, mode) + if a_db == 0: + raise error("Could not open file %s.db" % filename) + return dbm(a_db) From pypy-svn at codespeak.net Fri Jul 11 17:05:29 2008 From: pypy-svn at codespeak.net (Jackie Middleton) Date: Fri, 11 Jul 2008 17:05:29 +0200 (CEST) Subject: [pypy-svn] Dear pypy-svn@codespeak.net July 83% 0FF Message-ID: <20020101134121.4302.qmail@eel245.neoplus.adsl.tpnet.pl> An HTML attachment was scrubbed... URL: From adurdin at codespeak.net Fri Jul 11 17:14:18 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 17:14:18 +0200 (CEST) Subject: [pypy-svn] r56456 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711151418.CB00139B5A8@codespeak.net> Author: adurdin Date: Fri Jul 11 17:14:18 2008 New Revision: 56456 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) interp_itertools - tee() done, only one left! Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Fri Jul 11 17:14:18 2008 @@ -37,6 +37,7 @@ 'repeat' : 'interp_itertools.W_Repeat', 'starmap' : 'interp_itertools.W_StarMap', 'takewhile' : 'interp_itertools.W_TakeWhile', + 'tee' : 'interp_itertools.tee', } appleveldefs = { Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 17:14:18 2008 @@ -617,7 +617,7 @@ return self.space.call(self.w_fun, w_obj) -def W_StarMap___new__(space, w_subtype, w_fun,w_iterable): +def W_StarMap___new__(space, w_subtype, w_fun, w_iterable): return space.wrap(W_StarMap(space, w_fun, w_iterable)) W_StarMap.typedef = TypeDef( @@ -642,3 +642,73 @@ W_StarMap.typedef.acceptable_as_base_class = False +def tee(space, w_iterable, n=2): + """Return n independent iterators from a single iterable. + Note : once tee() has made a split, the original iterable + should not be used anywhere else; otherwise, the iterable could get + advanced without the tee objects being informed. + + Note : this member of the toolkit may require significant auxiliary + storage (depending on how much temporary data needs to be stored). + In general, if one iterator is going to use most or all of the + data before the other iterator, it is faster to use list() instead + of tee() + + Equivalent to : + + def tee(iterable, n=2): + def gen(next, data={}, cnt=[0]): + for i in count(): + if i == cnt[0]: + item = data[i] = next() + cnt[0] += 1 + else: + item = data.pop(i) + yield item + it = iter(iterable) + return tuple([gen(it.next) for i in range(n)]) + """ + if n < 0: + raise OperationError(space.w_ValueError, space.wrap("n must be >= 0")) + + tee_state = TeeState(space, w_iterable) + iterators_w = [space.wrap(W_TeeIterable(space, tee_state)) for x in range(n)] + return space.newtuple(iterators_w) +tee.unwrap_spec = [ObjSpace, W_Root, int] + +class TeeState(object): + def __init__(self, space, w_iterable): + self.space = space + self.w_iterable = self.space.iter(w_iterable) + self.num_saved = 0 + self.saved_w = [] + + def get_next(self, index): + if index >= self.num_saved: + w_obj = self.space.next(self.w_iterable) + self.saved_w.append(w_obj) + self.num_saved += 1 + return w_obj + else: + return self.saved_w[index] + +class W_TeeIterable(Wrappable): + def __init__(self, space, tee_state): + self.space = space + self.tee_state = tee_state + self.index = 0 + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + try: + w_obj = self.tee_state.get_next(self.index) + return w_obj + finally: + self.index += 1 + +W_TeeIterable.typedef = TypeDef( + '_tee', + __iter__ = interp2app(W_TeeIterable.iter_w, unwrap_spec=['self']), + next = interp2app(W_TeeIterable.next_w, unwrap_spec=['self'])) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 17:14:18 2008 @@ -379,7 +379,38 @@ it = itertools.starmap(bool, [0]) raises(TypeError, it.next) - + def test_tee(self): + import itertools + + it1, it2 = itertools.tee([]) + raises(StopIteration, it1.next) + raises(StopIteration, it2.next) + + it1, it2 = itertools.tee([1, 2, 3]) + for x in [1, 2]: + assert it1.next() == x + for x in [1, 2, 3]: + assert it2.next() == x + assert it1.next() == 3 + raises(StopIteration, it1.next) + raises(StopIteration, it2.next) + + assert itertools.tee([], 0) == () + + iterators = itertools.tee([1, 2, 3], 10) + for it in iterators: + for x in [1, 2, 3]: + assert it.next() == x + raises(StopIteration, it.next) + + def test_iterables_wrongargs(self): + import itertools + + raises(TypeError, itertools.tee, 0) + raises(ValueError, itertools.tee, [], -1) + raises(TypeError, itertools.tee, [], None) + + def test_iterables(self): import itertools @@ -396,6 +427,8 @@ itertools.repeat(None), itertools.starmap(bool, []), itertools.takewhile(bool, []), + itertools.tee([])[0], + itertools.tee([])[1], ] for it in iterables: @@ -403,7 +436,7 @@ assert iter(it) is it assert hasattr(it, 'next') assert callable(it.next) - + def test_docstrings(self): import itertools @@ -421,6 +454,7 @@ itertools.repeat, itertools.starmap, itertools.takewhile, + itertools.tee, ] for method in methods: assert method.__doc__ From antocuni at codespeak.net Fri Jul 11 17:23:28 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 11 Jul 2008 17:23:28 +0200 (CEST) Subject: [pypy-svn] r56457 - in pypy/branch/builtin-profiling/pypy/interpreter: . test Message-ID: <20080711152328.9909D169F52@codespeak.net> Author: antocuni Date: Fri Jul 11 17:23:26 2008 New Revision: 56457 Modified: pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py pypy/branch/builtin-profiling/pypy/interpreter/function.py pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Log: make profiling of builtin calls working also in the general case Modified: pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py Fri Jul 11 17:23:26 2008 @@ -694,17 +694,10 @@ return self.call_args(w_func, args) def call_valuestack(self, w_func, nargs, frame): - from pypy.interpreter.function import Function, Method - from pypy.interpreter.gateway import BuiltinCode + from pypy.interpreter.function import Function, Method, is_builtin_code if frame.is_being_profiled: ec = self.getexecutioncontext() - is_c_call = False - if isinstance(w_func, Method): - code = w_func.w_function.getcode() - elif isinstance(w_func, Function): - code = w_func.getcode() - if isinstance(code, BuiltinCode): - is_c_call = True + is_c_call = is_builtin_code(w_func) # XXX: this code is copied&pasted :-( from the slow path # below. The profiling info could be not very accurate Modified: pypy/branch/builtin-profiling/pypy/interpreter/function.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/function.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/function.py Fri Jul 11 17:23:26 2008 @@ -502,3 +502,11 @@ def descr_function_repr(self): return self.space.wrap('' % (self.name,)) + +def is_builtin_code(w_func): + from pypy.interpreter.gateway import BuiltinCode + if isinstance(w_func, Method): + code = w_func.w_function.getcode() + elif isinstance(w_func, Function): + code = w_func.getcode() + return isinstance(code, BuiltinCode) Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Fri Jul 11 17:23:26 2008 @@ -856,6 +856,7 @@ def call_function(f, oparg, w_star=None, w_starstar=None): from pypy.rlib import rstack # for resume points + from pypy.interpreter.function import is_builtin_code n_arguments = oparg & 0xff n_keywords = (oparg>>8) & 0xff @@ -865,7 +866,20 @@ arguments = f.popvalues(n_arguments) args = Arguments(f.space, arguments, keywords, w_star, w_starstar) w_function = f.popvalue() - w_result = f.space.call_args(w_function, args) + if f.is_being_profiled and is_builtin_code(w_function): + is_c_call = is_builtin_code(w_function) + ec = f.space.getexecutioncontext() + if is_c_call: + ec.c_call_trace(f, w_function) + try: + w_result = f.space.call_args(w_function, args) + except OperationError, e: + if is_c_call: + ec.c_exception_trace(f, e) + if is_c_call: + ec.c_return_trace(f, w_function) + else: + w_result = f.space.call_args(w_function, args) rstack.resume_point("call_function", f, returns=w_result) f.pushvalue(w_result) Modified: pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Fri Jul 11 17:23:26 2008 @@ -99,6 +99,7 @@ check_snippet('args = (1, 2); max(*args)') check_snippet('max(1, 2, **{})') check_snippet('args = (1, 2); max(*args, **{})') + check_snippet('abs(val=0)') def test_llprofile_c_exception(self): l = [] From arigo at codespeak.net Fri Jul 11 17:42:21 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 11 Jul 2008 17:42:21 +0200 (CEST) Subject: [pypy-svn] r56458 - in pypy/dist/pypy: module/itertools objspace/fake Message-ID: <20080711154221.89F1716A035@codespeak.net> Author: arigo Date: Fri Jul 11 17:42:19 2008 New Revision: 56458 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/objspace/fake/objspace.py Log: (adurdin, arigo) - fix(?) some detail in the fake object space - a comment about non-RPythonic usage; there is more to fix in interp_itertools. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 17:42:19 2008 @@ -476,6 +476,7 @@ if self.identity_fun: return self.space.newtuple(w_objects) else: + # XXX cannot use '*w_objects'; see space.call() return self.space.call_function(self.w_fun, *w_objects) @@ -712,3 +713,4 @@ '_tee', __iter__ = interp2app(W_TeeIterable.iter_w, unwrap_spec=['self']), next = interp2app(W_TeeIterable.next_w, unwrap_spec=['self'])) +W_TeeIterable.typedef.acceptable_as_base_class = False Modified: pypy/dist/pypy/objspace/fake/objspace.py ============================================================================== --- pypy/dist/pypy/objspace/fake/objspace.py (original) +++ pypy/dist/pypy/objspace/fake/objspace.py Fri Jul 11 17:42:19 2008 @@ -1,14 +1,15 @@ from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root +from pypy.interpreter.typedef import TypeDef from pypy.rlib.nonconst import NonConstant from pypy.rlib.rarithmetic import r_uint from pypy.rlib.rbigint import rbigint -class W_Type(W_Root): - _attrs_ = () +#class W_Type(W_Root): +# _attrs_ = () class W_Object(W_Root): _attrs_ = () -W_Object.typedef = W_Type() +W_Object.typedef = TypeDef('foobar') def make_dummy(a=W_Object(), b=W_Object()): def fn(*args): From antocuni at codespeak.net Fri Jul 11 18:21:56 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 11 Jul 2008 18:21:56 +0200 (CEST) Subject: [pypy-svn] r56460 - in pypy/branch/builtin-profiling/pypy/interpreter: . test Message-ID: <20080711162156.AAF4816A22B@codespeak.net> Author: antocuni Date: Fri Jul 11 18:21:54 2008 New Revision: 56460 Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Log: (antocuni, arigo around) make sure to trace c_calls when you call sys.setprofile in a function called from the current frame. The test is probably easier to understand than this sentence :-) Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py Fri Jul 11 18:21:54 2008 @@ -71,8 +71,7 @@ def leave(self, ec): self.framestack = ec.framestack self.w_tracefunc = ec.w_tracefunc - self.profilefunc = ec.profilefunc - self.w_profilefuncarg = ec.w_profilefuncarg + self.setllprofile(ec.profilefunc, ec.w_profilefuncarg) self.is_tracing = ec.is_tracing # the following interface is for pickling and unpickling @@ -106,20 +105,23 @@ def c_call_trace(self, frame, w_func): "Profile the call of a builtin function" - if self.profilefunc is not None: + if self.profilefunc is None: + frame.is_being_profiled = False + else: self._trace(frame, 'c_call', w_func) - #self._llprofile('c_call', w_func) def c_return_trace(self, frame, w_retval): "Profile the return from a builtin function" - if self.profilefunc is not None: + if self.profilefunc is None: + frame.is_being_profiled = False + else: self._trace(frame, 'c_return', w_retval) - #self._llprofile('c_return', w_retval) def c_exception_trace(self, frame, operationerr): "Profile function called upon OperationError." - if self.profilefunc is not None: - #self._llprofile('c_exception', operationerr) + if self.profilefunc is None: + frame.is_being_profiled = False + else: self._trace(frame, 'c_exception', operationerr) def _llprofile(self, event, w_arg): @@ -187,13 +189,15 @@ self.profilefunc = None self.w_profilefuncarg = None else: - self.w_profilefuncarg = w_func - self.profilefunc = app_profile_call + self.setllprofile(app_profile_call, w_func) def setllprofile(self, func, w_arg): self.profilefunc = func - if func is not None and w_arg is None: - raise ValueError("Cannot call setllprofile with real None") + if func is not None: + if w_arg is None: + raise ValueError("Cannot call setllprofile with real None") + for frame in self.framestack.items: + frame.is_being_profiled = True self.w_profilefuncarg = w_arg def call_tracing(self, w_func, w_args): Modified: pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/test/test_executioncontext.py Fri Jul 11 18:21:54 2008 @@ -124,3 +124,23 @@ check_snippet('d = {}; d.__getitem__(42)') + def test_c_call_setprofile_outer_frame(self): + space = self.space + w_events = space.appexec([], """(): + import sys + l = [] + def profile(frame, event, arg): + l.append(event) + + def foo(): + sys.setprofile(profile) + + def bar(): + foo() + max(1, 2) + + bar() + return l + """) + events = space.unwrap(w_events) + assert events == ['return', 'c_call', 'c_return', 'return', 'return'] From adurdin at codespeak.net Fri Jul 11 18:27:36 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Fri, 11 Jul 2008 18:27:36 +0200 (CEST) Subject: [pypy-svn] r56461 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080711162736.7DF0A16A22D@codespeak.net> Author: adurdin Date: Fri Jul 11 18:27:35 2008 New Revision: 56461 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin) - Removed cruft from test_itertools - Made interp_itertools more RPythonic Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Fri Jul 11 18:27:35 2008 @@ -382,30 +382,32 @@ i += 1 - self.iterators = iter(iterators_w) + self.iterators_w = iterators_w + self.current_iterator = 0 + self.num_iterators = len(iterators_w) self.started = False def iter_w(self): return self.space.wrap(self) def next_w(self): + if self.current_iterator >= self.num_iterators: + raise OperationError(self.space.w_StopIteration, self.space.w_None) if not self.started: - try: - self.w_it = self.iterators.next() - except StopIteration: - raise OperationError(self.space.w_StopIteration, self.space.w_None) - else: - self.started = True + self.current_iterator = 0 + self.w_it = self.iterators_w[self.current_iterator] + self.started = True while True: try: w_obj = self.space.next(self.w_it) except OperationError, e: if e.match(self.space, self.space.w_StopIteration): - try: - self.w_it = self.iterators.next() - except StopIteration: + self.current_iterator += 1 + if self.current_iterator >= self.num_iterators: raise OperationError(self.space.w_StopIteration, self.space.w_None) + else: + self.w_it = self.iterators_w[self.current_iterator] else: raise else: @@ -467,17 +469,16 @@ raise OperationError(self.space.w_StopIteration, self.space.w_None) try: - w_objects = [self.space.next(w_it) for w_it in self.iterators_w] + w_objects = self.space.newtuple([self.space.next(w_it) for w_it in self.iterators_w]) except OperationError, e: if e.match(self.space, self.space.w_StopIteration): self.iterators_w = None raise if self.identity_fun: - return self.space.newtuple(w_objects) + return w_objects else: - # XXX cannot use '*w_objects'; see space.call() - return self.space.call_function(self.w_fun, *w_objects) + return self.space.call(self.w_fun, w_objects) def W_IMap___new__(space, w_subtype, w_fun, args_w): @@ -545,7 +546,7 @@ self.space = space self.saved_w = [] self.w_iterable = space.iter(w_iterable) - self.saved_iterator = None + self.index = 0 self.exhausted = False def iter_w(self): @@ -556,10 +557,12 @@ if not self.saved_w: raise OperationError(self.space.w_StopIteration, self.space.w_None) try: - w_obj = self.saved_iterator.next() - except StopIteration: - self.saved_iterator = iter(self.saved_w) - w_obj = self.saved_iterator.next() + w_obj = self.saved_w[self.index] + except IndexError: + self.index = 1 + w_obj = self.saved_w[0] + else: + self.index += 1 else: try: w_obj = self.space.next(self.w_iterable) @@ -568,11 +571,12 @@ self.exhausted = True if not self.saved_w: raise - self.saved_iterator = iter(self.saved_w) - w_obj = self.saved_iterator.next() + self.index = 1 + w_obj = self.saved_w[0] else: raise else: + self.index += 1 self.saved_w.append(w_obj) return w_obj Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Fri Jul 11 18:27:35 2008 @@ -461,22 +461,23 @@ def test_subclassing(self): import itertools - # Although implemented as classes, the itertools functions should not be subclassable + # Although (mostly) implemented as classes, the itertools functions should not be subclassable iterables = [ - (itertools.chain, ()), - (itertools.count, ()), - (itertools.cycle, ()), - (itertools.dropwhile, (bool, [])), - (itertools.ifilter, (None, [])), - (itertools.ifilterfalse, (None, [])), - (itertools.imap, (None,)), - (itertools.islice, ([], 0)), - (itertools.izip, ()), - (itertools.repeat, (None,)), - (itertools.starmap, (bool, [])), - (itertools.takewhile, (bool, [])), + itertools.chain, + itertools.count, + itertools.cycle, + itertools.dropwhile, + itertools.ifilter, + itertools.ifilterfalse, + itertools.imap, + itertools.islice, + itertools.izip, + itertools.repeat, + itertools.starmap, + itertools.takewhile, + itertools.tee, ] - for cls, args in iterables: + for cls in iterables: try: class A(cls): pass @@ -484,4 +485,4 @@ pass else: assert False, "Subclassing should fail." - + From antocuni at codespeak.net Fri Jul 11 21:57:41 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 11 Jul 2008 21:57:41 +0200 (CEST) Subject: [pypy-svn] r56466 - pypy/branch/builtin-profiling/pypy/interpreter Message-ID: <20080711195741.6D7A016A163@codespeak.net> Author: antocuni Date: Fri Jul 11 21:57:40 2008 New Revision: 56466 Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Log: sometimes the annotator is smart enough to find bugs for you. Yaih static typing!:-) Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Fri Jul 11 21:57:40 2008 @@ -876,6 +876,7 @@ except OperationError, e: if is_c_call: ec.c_exception_trace(f, e) + raise if is_c_call: ec.c_return_trace(f, w_function) else: From arigo at codespeak.net Sat Jul 12 10:52:34 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 10:52:34 +0200 (CEST) Subject: [pypy-svn] r56477 - pypy/extradoc/sprintinfo/post-ep2008 Message-ID: <20080712085234.E55F2169EA6@codespeak.net> Author: arigo Date: Sat Jul 12 10:52:34 2008 New Revision: 56477 Modified: pypy/extradoc/sprintinfo/post-ep2008/planning.txt Log: Planning for the 3rd day of the sprint. Modified: pypy/extradoc/sprintinfo/post-ep2008/planning.txt ============================================================================== --- pypy/extradoc/sprintinfo/post-ep2008/planning.txt (original) +++ pypy/extradoc/sprintinfo/post-ep2008/planning.txt Sat Jul 12 10:52:34 2008 @@ -13,30 +13,29 @@ * Django seems to be running. * Fixed _rawffi for Mac OS X * CherryPy works out of the box + * Pinax project/application runs on pypy-c + * update play1.codespeak.net and make this work there + * filed even more Django tickets + * started http://code.djangoproject.com/wiki/DjangoAndPyPy - - support for profiling the calls to builtins (IN-PROGRESS) - (stephan, anto) + - support for profiling the calls to builtins with + sys.setprofile(): TESTING - fix the default options of "translate.py", make "--help" show only the reasonable options, remove "--faassen", etc. (samuele, others for discussion) - - dotviewer improvements (expanding & collapsing, etc.) - and eventually make it a separate project - * Having fun - * Minor progress - * IN-PROGRESS (dinu, marius) - - start a tutorial on how to write a language interpreter - in RPython with PyPy + in RPython with PyPy: SOME PROGRESS, but no more today - - port the stackless transform to ootypesystem - - - improve rlib.rjvm (java bindings for rpython) - - - profile & speedup pypy-jvm - - more extension modules, with ctypes or rpython - * itertools IN-PROGRESS (adurdin, jakub, anto around) + * itertools EVEN MORE IN-PROGRESS (adurdin, jakub) + * hashlib, ctypes->openssl IN-PROGRESS (gasper, fijal + around) + + - port the Python interpreter for Malbolge to RPython: + IN-PROGRESS + + - work on _rawffi.py for Jython was started :-) - general wizardry (anto, arigo) From arigo at codespeak.net Sat Jul 12 11:17:21 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 11:17:21 +0200 (CEST) Subject: [pypy-svn] r56478 - pypy/dist/pypy/config Message-ID: <20080712091721.AB40A169EC7@codespeak.net> Author: arigo Date: Sat Jul 12 11:17:21 2008 New Revision: 56478 Modified: pypy/dist/pypy/config/pypyoption.py Log: The dependency rctime->select was removed some time ago. Modified: pypy/dist/pypy/config/pypyoption.py ============================================================================== --- pypy/dist/pypy/config/pypyoption.py (original) +++ pypy/dist/pypy/config/pypyoption.py Sat Jul 12 11:17:21 2008 @@ -44,8 +44,6 @@ # because 'P' is missing from the app-level one '_rawffi': [("objspace.usemodules.struct", True)], } -if os.name == "posix": - module_dependencies['rctime'] = [("objspace.usemodules.select", True),] module_import_dependencies = { # no _rawffi if importing pypy.rlib.libffi raises ImportError From antocuni at codespeak.net Sat Jul 12 11:33:44 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Sat, 12 Jul 2008 11:33:44 +0200 (CEST) Subject: [pypy-svn] r56479 - pypy/branch/builtin-profiling/pypy/interpreter Message-ID: <20080712093344.9DF5A39B5BA@codespeak.net> Author: antocuni Date: Sat Jul 12 11:33:42 2008 New Revision: 56479 Modified: pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py pypy/branch/builtin-profiling/pypy/interpreter/function.py pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Log: rpython fixes Modified: pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/baseobjspace.py Sat Jul 12 11:33:42 2008 @@ -714,7 +714,7 @@ return w_res except OperationError, e: if is_c_call: - ec.c_exception_trace(frame, e) + ec.c_exception_trace(frame, e.w_value) raise finally: if isinstance(args, ArgumentsFromValuestack): Modified: pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/executioncontext.py Sat Jul 12 11:33:42 2008 @@ -117,12 +117,12 @@ else: self._trace(frame, 'c_return', w_retval) - def c_exception_trace(self, frame, operationerr): + def c_exception_trace(self, frame, w_exc): "Profile function called upon OperationError." if self.profilefunc is None: frame.is_being_profiled = False else: - self._trace(frame, 'c_exception', operationerr) + self._trace(frame, 'c_exception', w_exc) def _llprofile(self, event, w_arg): fr = self.framestack.items Modified: pypy/branch/builtin-profiling/pypy/interpreter/function.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/function.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/function.py Sat Jul 12 11:33:42 2008 @@ -506,7 +506,11 @@ def is_builtin_code(w_func): from pypy.interpreter.gateway import BuiltinCode if isinstance(w_func, Method): - code = w_func.w_function.getcode() + w_f = w_func.w_function + assert isinstance(w_f, Function) + code = w_f.getcode() elif isinstance(w_func, Function): code = w_func.getcode() + else: + code = None return isinstance(code, BuiltinCode) Modified: pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py (original) +++ pypy/branch/builtin-profiling/pypy/interpreter/pyopcode.py Sat Jul 12 11:33:42 2008 @@ -875,7 +875,7 @@ w_result = f.space.call_args(w_function, args) except OperationError, e: if is_c_call: - ec.c_exception_trace(f, e) + ec.c_exception_trace(f, e.w_value) raise if is_c_call: ec.c_return_trace(f, w_function) From pedronis at codespeak.net Sat Jul 12 11:43:26 2008 From: pedronis at codespeak.net (pedronis at codespeak.net) Date: Sat, 12 Jul 2008 11:43:26 +0200 (CEST) Subject: [pypy-svn] r56480 - pypy/tag/run-danjgo Message-ID: <20080712094326.DBEB239B592@codespeak.net> Author: pedronis Date: Sat Jul 12 11:43:24 2008 New Revision: 56480 Added: pypy/tag/run-danjgo/ - copied from r56444, pypy/dist/ Log: tagging a known to work rev for running django From pedronis at codespeak.net Sat Jul 12 11:44:06 2008 From: pedronis at codespeak.net (pedronis at codespeak.net) Date: Sat, 12 Jul 2008 11:44:06 +0200 (CEST) Subject: [pypy-svn] r56481 - pypy/tag/run-danjgo Message-ID: <20080712094406.8725439B5BB@codespeak.net> Author: pedronis Date: Sat Jul 12 11:44:05 2008 New Revision: 56481 Removed: pypy/tag/run-danjgo/ Log: kill From pedronis at codespeak.net Sat Jul 12 11:46:08 2008 From: pedronis at codespeak.net (pedronis at codespeak.net) Date: Sat, 12 Jul 2008 11:46:08 +0200 (CEST) Subject: [pypy-svn] r56482 - pypy/tag/run-django Message-ID: <20080712094608.54B2139B593@codespeak.net> Author: pedronis Date: Sat Jul 12 11:46:07 2008 New Revision: 56482 Added: pypy/tag/run-django/ - copied from r56444, pypy/dist/ Log: tag revision known to work to run django From adurdin at codespeak.net Sat Jul 12 11:54:50 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Sat, 12 Jul 2008 11:54:50 +0200 (CEST) Subject: [pypy-svn] r56483 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080712095450.E17EF39B593@codespeak.net> Author: adurdin Date: Sat Jul 12 11:54:50 2008 New Revision: 56483 Modified: pypy/dist/pypy/module/itertools/__init__.py pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: (adurdin, jlg) - interp_itertools: groupby implemented, but not yet correct - interp_itertools tests on exception string only look for the important data Modified: pypy/dist/pypy/module/itertools/__init__.py ============================================================================== --- pypy/dist/pypy/module/itertools/__init__.py (original) +++ pypy/dist/pypy/module/itertools/__init__.py Sat Jul 12 11:54:50 2008 @@ -29,6 +29,7 @@ 'count' : 'interp_itertools.W_Count', 'cycle' : 'interp_itertools.W_Cycle', 'dropwhile' : 'interp_itertools.W_DropWhile', + 'groupby' : 'interp_itertools.W_GroupBy', 'ifilter' : 'interp_itertools.W_IFilter', 'ifilterfalse' : 'interp_itertools.W_IFilterFalse', 'imap' : 'interp_itertools.W_IMap', Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Sat Jul 12 11:54:50 2008 @@ -718,3 +718,142 @@ __iter__ = interp2app(W_TeeIterable.iter_w, unwrap_spec=['self']), next = interp2app(W_TeeIterable.next_w, unwrap_spec=['self'])) W_TeeIterable.typedef.acceptable_as_base_class = False + + +class W_GroupBy(Wrappable): + + def __init__(self, space, w_iterable, w_fun): + self.space = space + self.w_iterable = self.space.iter(w_iterable) + self.identity_fun = self.space.is_w(w_fun, self.space.w_None) + self.w_fun = w_fun + self.index = 0 + self.lookahead = False + self.exhausted = False + self.started = False + self.group_edge = True + self.w_lookahead = self.space.w_None + self.w_key = self.space.w_None + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if self.exhausted: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + if not self.started: + self.started = True + try: + w_obj = self.space.next(self.w_iterable) + except OperationError, e: + if e.match(self.space, self.space.w_StopIteration): + self.exhausted = True + raise + else: + self.w_lookahead = w_obj + if self.identity_fun: + self.w_key = w_obj + else: + self.w_key = self.space.call_function(self.w_fun, w_obj) + self.lookahead = True + + if not self.group_edge: + # Consume unwanted input until we reach the next group + try: + while True: + self.group_next(self.index) + except StopIteration: + pass + if self.exhausted: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + w_iterator = self.space.wrap(W_GroupByIterator(self.space, self.index, self)) + return self.space.newtuple([self.w_key, w_iterator]) + + def group_next(self, group_index): + self.group_edge = False + if group_index < self.index: + raise StopIteration + else: + if self.lookahead: + self.lookahead = False + return self.w_lookahead + + try: + w_obj = self.space.next(self.w_iterable) + except OperationError, e: + if e.match(self.space, self.space.w_StopIteration): + self.exhausted = True + raise StopIteration + else: + raise + else: + if self.identity_fun: + w_new_key = w_obj + else: + w_new_key = self.space.call_function(self.w_fun, w_obj) + if self.space.eq_w(self.w_key, w_new_key): + return w_obj + else: + self.index += 1 + self.w_lookahead = w_obj + self.w_key = w_new_key + self.lookahead = True + self.group_edge = True + raise StopIteration + +def W_GroupBy___new__(space, w_subtype, w_iterable, w_fun=None): + return space.wrap(W_GroupBy(space, w_iterable, w_fun)) + +W_GroupBy.typedef = TypeDef( + 'groupby', + __new__ = interp2app(W_GroupBy___new__, unwrap_spec=[ObjSpace, W_Root, W_Root, W_Root]), + __iter__ = interp2app(W_GroupBy.iter_w, unwrap_spec=['self']), + next = interp2app(W_GroupBy.next_w, unwrap_spec=['self']), + __doc__ = """Make an iterator that returns consecutive keys and groups from the + iterable. The key is a function computing a key value for each + element. If not specified or is None, key defaults to an identity + function and returns the element unchanged. Generally, the + iterable needs to already be sorted on the same key function. + + The returned group is itself an iterator that shares the + underlying iterable with groupby(). Because the source is shared, + when the groupby object is advanced, the previous group is no + longer visible. So, if that data is needed later, it should be + stored as a list: + + groups = [] + uniquekeys = [] + for k, g in groupby(data, keyfunc): + groups.append(list(g)) # Store group iterator as a list + uniquekeys.append(k) + """) +W_GroupBy.typedef.acceptable_as_base_class = False + +class W_GroupByIterator(Wrappable): + def __init__(self, space, index, groupby): + self.space = space + self.index = index + self.groupby = groupby + self.exhausted = False + + def iter_w(self): + return self.space.wrap(self) + + def next_w(self): + if self.exhausted: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + + try: + w_obj = self.groupby.group_next(self.index) + except StopIteration: + self.exhausted = True + raise OperationError(self.space.w_StopIteration, self.space.w_None) + else: + return w_obj + +W_GroupByIterator.typedef = TypeDef( + '_groupby', + __iter__ = interp2app(W_GroupByIterator.iter_w, unwrap_spec=['self']), + next = interp2app(W_GroupByIterator.next_w, unwrap_spec=['self'])) +W_GroupByIterator.typedef.acceptable_as_base_class = False Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Sat Jul 12 11:54:50 2008 @@ -245,17 +245,18 @@ raises(StopIteration, it.next) def test_chain_wrongargs(self): - import itertools + import itertools, re raises(TypeError, itertools.chain, None) raises(TypeError, itertools.chain, [], None) - + + # The error message should indicate which argument was dodgy for x in range(10): args = [()] * x + [None] + [()] * (9 - x) try: itertools.chain(*args) except TypeError, e: - assert str(e) == "chain argument #%d must support iteration" % (x + 1) + assert re.search(r'\b%d\b' % (x + 1), str(e)) else: fail("TypeError expected") @@ -332,17 +333,18 @@ assert it1.next() == 5 def test_izip_wrongargs(self): - import itertools + import itertools, re # Duplicate python 2.4 behaviour for invalid arguments raises(TypeError, itertools.izip, None, 0) + # The error message should indicate which argument was dodgy for x in range(10): args = [()] * x + [None] + [()] * (9 - x) try: itertools.izip(*args) except TypeError, e: - assert str(e) == "izip argument #%d must support iteration" % (x + 1) + assert re.search(r'\b%d\b' % (x + 1), str(e)) else: fail("TypeError expected") @@ -403,13 +405,79 @@ assert it.next() == x raises(StopIteration, it.next) - def test_iterables_wrongargs(self): + def test_tee_wrongargs(self): import itertools raises(TypeError, itertools.tee, 0) raises(ValueError, itertools.tee, [], -1) raises(TypeError, itertools.tee, [], None) + def test_groupby(self): + import itertools + + it = itertools.groupby([]) + raises(StopIteration, it.next) + + it = itertools.groupby([1, 2, 2, 3, 3, 3, 4, 4, 4, 4]) + for x in [1, 2, 3, 4]: + k, g = it.next() + assert k == x + assert len(list(g)) == x + raises(StopIteration, g.next) + raises(StopIteration, it.next) + + it = itertools.groupby([0, 1, 2, 3, 4, 5], None) + for x in [0, 1, 2, 3, 4, 5]: + k, g = it.next() + assert k == x + assert g.next() == x + raises(StopIteration, g.next) + raises(StopIteration, it.next) + + it = itertools.groupby([0, 0, 0, 0, 1]) + k1, g1 = it.next() + k2, g2 = it.next() + raises(StopIteration, g1.next) + assert g2.next() == 1 + raises(StopIteration, g2.next) + + def half_floor(x): + return x // 2 + it = itertools.groupby([0, 1, 2, 3, 4, 5], half_floor) + for x in [0, 1, 2]: + k, g = it.next() + assert k == x + assert half_floor(g.next()) == x + assert half_floor(g.next()) == x + raises(StopIteration, g.next) + raises(StopIteration, it.next) + + # Grouping is not based on key identity + class NeverEqual(object): + def __eq__(self, other): + return False + objects = [NeverEqual(), NeverEqual(), NeverEqual()] + it = itertools.groupby(objects) + for x in objects: + print "Trying", x + k, g = it.next() + assert k is x + assert g.next() is x + raises(StopIteration, g.next) + raises(StopIteration, it.next) + + # Grouping is based on key equality + class AlwaysEqual(object): + def __eq__(self, other): + return True + objects = [AlwaysEqual(), AlwaysEqual(), AlwaysEqual()] + it = itertools.groupby(objects) + k, g = it.next() + assert k is objects[0] + for x in objects: + assert g.next() is x + raises(StopIteration, g.next) + raises(StopIteration, it.next) def test_iterables(self): import itertools @@ -419,6 +487,7 @@ itertools.count(), itertools.cycle([]), itertools.dropwhile(bool, []), + itertools.groupby([]), itertools.ifilter(None, []), itertools.ifilterfalse(None, []), itertools.imap(None), @@ -446,6 +515,7 @@ itertools.count, itertools.cycle, itertools.dropwhile, + itertools.groupby, itertools.ifilter, itertools.ifilterfalse, itertools.imap, @@ -467,6 +537,7 @@ itertools.count, itertools.cycle, itertools.dropwhile, + itertools.groupby, itertools.ifilter, itertools.ifilterfalse, itertools.imap, From adurdin at codespeak.net Sat Jul 12 11:58:36 2008 From: adurdin at codespeak.net (adurdin at codespeak.net) Date: Sat, 12 Jul 2008 11:58:36 +0200 (CEST) Subject: [pypy-svn] r56484 - in pypy/dist/pypy/lang/malbolge: . examples Message-ID: <20080712095836.CFDB839B5BD@codespeak.net> Author: adurdin Date: Sat Jul 12 11:58:35 2008 New Revision: 56484 Added: pypy/dist/pypy/lang/malbolge/ (props changed) pypy/dist/pypy/lang/malbolge/examples/ pypy/dist/pypy/lang/malbolge/examples/99bottles.mbs (contents, props changed) pypy/dist/pypy/lang/malbolge/examples/copy.mbs (contents, props changed) pypy/dist/pypy/lang/malbolge/examples/hello-world.mb pypy/dist/pypy/lang/malbolge/examples/hello-world.mbs (contents, props changed) pypy/dist/pypy/lang/malbolge/malbolge.py (contents, props changed) Log: (adurdin) Ported Toni Ruotto's Malbolge interpreter to RPython: hello_world runs, but bugs or missing implementation prevents the rest from working. Added: pypy/dist/pypy/lang/malbolge/examples/99bottles.mbs ============================================================================== --- (empty file) +++ pypy/dist/pypy/lang/malbolge/examples/99bottles.mbs Sat Jul 12 11:58:35 2008 @@ -0,0 +1,252 @@ +#!/usr/bin/env malbolge +# +# 99bottles, written by Hisashi Iizawa in 2005 +# +# Prints out lyrics for '99 Bottles of Beer'. + +b'`;$9!=IlXFiVwwvtPO0)pon%IHGFDV|dd at Q=+^:('&Y$#m!1S|.QOO=v('98$65aCB}0i.Tw+QPU'7qK#I20jiDVgG +S(bt<%@#!7~|4{y1xv.us+rp(om%lj"ig}fd"cx``uz]rwvYnslkTonPfOjiKgJeG]\EC_X]@[Z7~;:9y16w43s10)p-,l*#(i&%e#d!~``{tyxZpuXsrTTongOkdMhg`Hd]ba`_^W@[ZYXW9UNSRQPOHMLK +J-++FE''=<;:387xw43s10/(-&m*)('&}${d!~}|^zyxwvutmVqpiRQlkjiKafedc\E`_^@\[ZYX;V9NMRQ42NG +LK.IH*F?DCBA$#>7~;{{8xx5uu2rr/oo,ll)ii&f|e"!aw`{z\r[vXnmVTpongPkNihgJ_dcFa`B^]\UZ=RWV8TSLQ4O +N0LE.IHA)E>'BA:?!7~5|38y6/v321q).-&m*)i'&%|{d!~}_{zs\wvutsUqTonPlOjiKgJedFbE`_A]@[Z7~;:987w5v32r0)p-,+k)('~g$#"b~w|uz]xwvutsrqTinQlOjLhgfeH]bE`CB]\>ZSXWVUTS +RQPON1LE.I,+*((&&$$""~~||zzxxv4u210/(-n+l)(i&g$ddy~}`u^]\ZvutVlUjSQQOOdMKgfeG]F[DBB@@>><<:VU +T6L5JO200EJ-HG*E>'B%$9>=<|4{2y05v321r).o,mlj(igg|#d!~}`uz]x[ZotWUUjoRmlkNibKJIGGEEZ_B]\?Z=XW +PU876442NM/KD-B+))''%%##!!}}{{yyw5v32s0q.-&+l)j'hff{"caav{^yxwZutslUpSnQOOdiLgfHHcba`Y^A\?Z= +;;PU8SRQ4ONMLEJ-,+))''%%##!=<;{3z1xvvttrrppnnll#j!&g$#d!b}|{zyr[vYtsrTjShQfkNihgJedcba`Y^A\? +Z=;WV9TSRQPOHM0K.-++)ED&B;$9"~<;:z2y0wuussqqoom+ljj!&%$dzcx}`{zy\wvutsrqjSnQPNNLhgIedG\EZCA] +\[=SY<;P9775533H1/KJ,HA*?(&&$$">=<|4{2ywwu321q)p'nl*k('gg${"c~a`^z]xwvYtmrUpSRPlOMMbK`IGGEE +Z_^]?U>S<::8866442200.JIH*@)>C&A@?"=<5|{8y65vtt10/(-n+lk"'&%e{dyb``^^\\ZvutVlUjSQmlkMcLaJHHF +bECCX]\[=S>>YXWP9T76K42200.JI+G@)>'%A@?!7~5|zzx654t,s*qo-n+*jj!h%fec!b}|{^s\[vYWWlqTonQlO +jchKfIHFFDDB^]\>T=R;P9NS6QPO2MLE.-,*FED&<%:#!!}}{{yyw543s+r)pnnl*kii~%f#"caa|{zsx[ZutVrkTinQ +lkNiLgfe^cFEDYBW\[=YR;P977553O200EJIH*@)>C&$$9>!<;|9z7654-tsrppnnll#('&f|ezca}|{]s\qZXtsrTjS +hQOOMihgI_H]FDDB^A\[><=<5:{zy0wuussqqoomm$ki'hff{"c~}`{t]\wvuWmVk +pSnmPNNcLKfIGG\aD_^A\?T=<;99775QPO1G0E.,HG)E>'<%#?"~~5:98x0w.ussq/pnn%*k('hff#z!ba|{z\r[puXs +rUpSnglONihgI_H]FDDYBW\[Z=}5|3zxxv4u21r/p-,+*#(i&g$# +c!~av_t][[YutsUkTiRPPNNLLJJHHFFDDB^A\[Z=XWVUTM6Q43HM0..CH+FE''BA@?>=6;|9z765u-t+0q.-,m*)('&% +|#dcb``uzy[wpYnWUUSonmOeNcLJJHHFFDDBB@\?ZY=}5|3zxxvvttrrppn,m*)ii +&%$#"!~}v{^y\wvuWmVkpSnmlOjihgfedc\aD_BAV[ZC&%$""7~|:9y70w.us10/o' +n%ljjhhffddb~a|{^\\wvutmVUTRnQlkNiLgfed]FE`_A]V?TY=<;:927x5vuss +qqo-,+k#j!hffddbb``^^s\qZXXVVTpSQQfkNihg`IHcbaCYBW\?==RW:UT755J321FK.IH+F)>'B%$9"~~||zzxxvvt +210p(o&mkki'&%e{dyb``^z]xwvYtmrUTSQmlkMcLaJHdcbDZCXA?[><T=R;99775Q4ON00KJIBG*E('%A$?>=~;:927 +xwvttr0/.n,m$)('g}f{"c~}`{^yxwvoXsVUSShQOkjLhaJ_HFba`BXAV?=YX:VO8M644220L/JI++FEDCB;@#>!<;:z +2y05v321r/.-,+$)j'hgeeccaa__]][wZXXmVkTiRPPNjMhgfIdcba`_XA@?==;;9977L5J31MLK-C,A*((&BA@"8!6} +{{y765u-t+rppn,m*)j'h%$#"!~}v{^y\[YutsUkTiRPPNNLhgfH^G\ECCAA??==;;9UTS5K4IN1LKJC,G*)''%%#?>= +}5|3zxxvvttrrppnnl*k('gg|e"c~a__ty\wvuXmVUTinmlNdMbKIedFb[DYB@\?==RWVU7M6K42N1LK.,,G at E('&$@# +>=~;|927xw4uss*/.n,%l#(i&%f#d!~w`{^][[YYWWlUjonmOeNcLJJHHFFDDB^A\[==XWVOT7R542N1LKJ-HGF?D'&% +:?"=<}:3z7xwuussqqo-,+k#j!h}fddb~}_{t]r[YYWWUqpoQgPeNLLJJHHFFDDBB@@>Z=XWV9N7R54I2GLKJ,B+@)'' +%%#?>=}5|3zx654t,s*qoommkkiig%f#"bb}v{^y\wvuWmVkpSnmlOjchKJIGGEaD_^A\?ZYR;V986RQP2H1F/--++)) +''%%##!=~;:zz765.3t1rq(-,+k#j!&g$#"c~}|uz]x[ZXtsrTjShQOOMMKKIIGGEECCAA?[><=6}|9zxx/432r*q(o&mk)j'&g$e"!~}|{zsx[vYXVVTponPfOdMbKIIGG\ECCAA??=YXW9O8MR5PONG0K +.I,**?D'BA at 9"=~}4{yywwuussqqoommkki'h%$ddyb}`_ty\ZZotsrTjShmPkjiLaJIHFba`BXAV?==;;9977553311 +//-IHG)?(=B%@?"=~;49z7x543s+r)pnnl*)(h~g|ec!b``uzyxZpYnWUqpoQgPeNLLJJHHFFD`C^]??ZYR;V986R5PO +N1LKD-,+))''%%##!=<;{3z1xvvttrrppn,mkk"'&f${dy~a|{^\\wvunsVUpSQQfkNihKfIdcb[`C^A\[ZZS7~;|{y76v4-t+rp.-,l$k"iggeecca}`{z\\wvunsVqTonPleNchKfedGba`Y^A@?==;W +VU7M6K42N1//DIHG)?(=&$$""~~||zzxxv4u21rpp-,+*#ji&geez!b}|_z]xwvunWrUpSQQfkjiKaJ_HFbE`_AA\[ZY +XQV9T764P3NML/JIHGF?D'&%##!=<;{3z1x/v-2s0/p-n+*)('&}f#d!b``uz][[pYnsrqSiRgPNNLLJfedF\EZCA]\[ +=S!<}:{yy05v321r/.-,+*)"'hg$e"!b``{zyxwvutmVqToRPlkNihgfedcb[DCBW +@>><<::8TSR4J3H1//--++))''%A@?!7~5:{87x5.u2s0/o-&m$)j'&ff{"c~a|_]]rwZutsVkpSnQPeNchgfH^G\ECC +AA??==;;997S6QPO2MF/J-,A*((&&$$""~~||zzxx/4u210q.-&+l)jig%$d"ybw|_zyx[vunsVUTRnmlNdMbKIIGGEa +`B^W at U><<::88664PON0F/DI,GF)D'BA at 9"=~}4{yy0wuus10/o'n%ljjh&%$dzcxa__]][wvXtmVkTRRPPNjMhgIIdc +baZ_B]@[ZY;Q:OT7RQP3NMLKDI,+FED&<%:#!=<|:3z1xvvttrrppnnlljjh&g$#d!b}|{zyr[vYXVrqpRhQfOMMKKII +GGEECCAA??=Y'=~;:9876/4utsq/pnn%*)i'~g|ec!b}|_z]xwvutsrkToRQOkjiKa +J_HFFDDBB@\?==RWV8TM6K42NML.D-B+))'C&A@""=<;:9876/4u2s0/o-&m$)j'&%f#"!~}|{zsx[vuXsVqSShmlOdM +hKJ_HFFDDBB@\[ZT=RW:UT7R5J32M0KJ-++F?D'&A@">7~5:{87xvv32+rq.omm$)j'&g$e"!xa|_^\xwYunWlUSonm +OeNcLJJHHFFDDBB@@>Z=XW99TSRKP3N10.J-HGF)DCB;@#"!6}{{y765u-t+rppn,+*j"i~geeccaa__]][wZutWUUpo +nmfONihgI_H]bE`_B]@[ZYXQ:9T755JO200E.CHGF(>'!<}:9876/4utsq/.-m%l#jhhf$#c!xav_]][[YuXsrU +pSnmlkjchKJIGGEa`_AW at U><<::8TS5QJ3H1/KJI+A*?(&&$@#>=~||9876543,1rq.-m+$k"'h%$e"c~}|{zyxqvYXs +VTTinmlNdMbgJedGbE`_^]\[ZYR;:9775QPO1M0EJ-HG*E>'B%$9"~~||zzxxvvttrrppn,m*)ii~%f#d!b``uz]xwvY +nsVqTSQQOkjiKaJ_HFFDDBB@@>><<::886R5PON1LE.-H+))>'Y<;P977553311//--++)EDC%;$9"7~5:{876w432+rqpnn%lj('g%|ezcaa__]y +\wvYtWrqpohmPkNMKKIIGGEECCAA??==;W:88MR5PON1LKJIHA*)(=&$$">!<;|zz765432+0qpo&+*)i!h}fd"!~`v_ +t][[YutsUkTiRPPNNLLJJHdGbaD_B]\[ZYXQV9T76KP3NM//JIHGFED=&A$#8=~||3876v.u,1r/.-n+*)('&%|edcx} +|^zs\qZXXVrqSohQfkNihKf_HGF[`_^@V?TY<::OT7RQ4OH10K.IH+F)>'B%@#!=<}4{z765u-t+r).o,+l)j'~%fedb +~a__ty\wvYtWrkpSRQfkNLLafIdcFaD_^W@?Z=;;PU8SR533NMLEJ-,G*((=B%@?"=~;:927xw43s10q(-,+k#j!hffd +db~}_{t]r[YYWWUUSSQQOkNihKfIdcbaZCBA?[><><<:V9TSR5PONMLKD-,+@)>C&$$9>!<;|92yx543s+r)p'n%lj(i +&%f#zc~a`u^\\ZZXXVVTTRRPPNNLLafIdcbEZ_B]@?=Y<::OT7RQP3HM0K.-BG*((=B%@?>!<5|{z1xvvt2sqq(-,+k# +j!hffddbb``^^\\ZvuWslUjoRmlOMMhg`eHGbECCX]@[Z=X;VUNS6Q4O200EJ-++ at E(CBA$?>=6}:{8yww.3t10pp-,+ +*#(i&g$eccx}`{zy\wvutmrUpSnQOOdiLJJ_dGba`C^]\[ZSC&A$?"~~5:{876w43210/(- +n+l)jhh}$eccx}`{zy\wvutsrqjSnQlOMMbgJedFFa`_^]\[ZSX;V9T755JO2MLK.IHGFEDCB;@#"!6}{{y765u3t+0q +.-n+$k(i&geez!b``uz]xwvYnWrUpSQQfkNihJJe^cFaD_B@@UZ=XWV9TMR5P3200..,,**((&&$$""~<;:z2y0w.3t1 +0/p-,%l)jiggeecca}|{]s\qZXXVVTponPfOdMKgfeG]F[DBB@@>><<:V9TS55PONGL/J-H+))>C&A@?"=<;49zy6w43 +tr*qp-n+*kii~g$e"ca}|_t]x[vuWslUjoRmlNNibgJeHcFDDY^A\[Z=XQV9T76K4INML.D-B+))''%%##!!}}{9z765 +v32+rqp'nllj('&f|ezca}|{]s\qZXtsrTjShmPkjMKKfed]bEDCA]@[Z=X;VUTMR5P320LKJ,B+@)''%%##!!}}{{yy +wwu321q)p',m*)(i&%$#zc~a|{]yr[pYWsVqpohQlOjMKK`I^cbD`YBW\?ZYX;PU8S653311//--+GFE'=&;$""~~|:9 +y70w.ussqqo-n+*)j!&g$e"!~`v{zyx[voXWrUpoRmPkdMhKJHdGEEZ_^]?U>S<:VUT6L5J311//--++))''%A$?>~~; +:38y6wvt2s0/.o,+$)j'hgeeccaa_{zy[qZoXmVTTRRPPeNLLJJHdGba`C^]\U>=X;VU86L5P32GLKJ,B+@)''%%##!= +~||3876v.u,sqqo-,+k#j!&g$#ccx}`{^y\ZZotWrqpShmPkNMKK`edFb[DYB@@>><<::88664PON0F/D-BG*EDC&A:# +"!}}{{yyw543s+r)p'n%*k('h%f#"y~a|_^\\ZZXXmrqpRhQfOMiLJJ_dcEaZCXA?[ZY;Q:O8664PO1MF/DI,GFE(CBA +:#"=~||38y65v3,srq(-n+*k(i~%f#dcaav_t][wvuWmVkTRRPPNNLLJJHHFFD`CAAV[>YXW:UN7R542N1//D-BG*((= +B%@?>!<;49z7x5vtt+r).-,l$k"'h%$#d!~w|_z]\ZZoXVrqSohQfOMMKgfeG]F[DBB@@>ZYX:P9N75QPO1G0E.,,**( +(&B%@?!!<;:3z7xwu3t10/p-,+$kj'h%$ecc~}|{ty\wZutVrkTinQlkMMhgfed]FaD_B@@UZ=XWV9TSRQPI21LKJ,B+ +@)''%%##!!}}{{yyw5v32sqq.-,+*)"'hg$#"bxav{^yx[vYtsrqpohmPONchKII^cFa`C^W@?Z=;;PU8SR5PI2M0/D- +++)EDC%;$9"~~||z876v.u,sqqoommk)j'&ff{d!ba_{^yxwZoXWVkTinmlNdMbgJedGbE`Y^A@?=YXW9O8M6442NM/K +D-B+))''%A$?>!<}:38y6wv-21q/(o&mk)(h&}f{dbb``^^\\ZZXXVVTpSnmOOjihafIdGFD`C^]\?ZYXQV9T7R533HM +LK-C,AF)DCB%@?>=6}|{2ywwuuss*q(om+*)i!h}fddbb``^^\\ZvYtsVTTonmlkdiLKJHdGbaD_B]\[ZYRW:U87L5J3 +H1/KJI+A*?D'BA at 9"!<}{{2765u-t+0q.-n+l#jih}f{dy~a|{^y\qZuXWlUjonmOeNcLJfIdcbE`Y^A\?><<::88664 +PON0F/D-++))'CB$@9"7~||zzx6w432s0).onmk)(h&}f{db~}|^t]r[YYWWUUSSQQOOMMKgJHH]bE`_B]@[ZS!<;:38y6w432r*/.-,m*)('~gf#d!~a|_zyxwpYXWUqpoQgPeNLLJJHHFFDDBB@@UZ=XW:U +8SRQPOHM0/.,HG)E>'<%#?"~~5:98x0w.ussq/p-,m*k('&%${"c~a`u^\\qZotWrqSSnmlkjibKfIdcbDZCX]@[ZYT= +RW:UT7R5JO21L/JI,G*E>'&A$?>!<}:3z7xwu321q)p'nlljjhhffddbb``^zyxZpYnsVqpRRmlejMhKfIGG\aD_^]@[ +ZSX;:977553311//--++)E(&&;@?>~6}49z76w4u210)po,mkk"'h%$ecc~}|{ty\[vuWslUjoRmlOjMhgfe^cFaDCXA +?[><YXWVOT7R542200..,HGF(>'<%:#8!}}{987w/v-2sqq(-n+*)j'&%$#zcb}`{z]x[vutsrqjoRQlOjiLgJe +dcba`Y^A@?=YXW9O8M6K4I20L/--BGFE'=&;@#>=~;|9876543,s0qpnnlljjhhffddbb`|_]]rwZutVVqponmlkjchK +fIdcbDZCX]@[ZY'<%:#!!}}{{2ywwuussq/pnn%*k('&}f#d!b``uz]xwvoXsVUjonPlkNc +LafedF\EZCAA??==;;997SRQ3I2G0..,H+FED'<%$#!!};:9y1x/vttrrppnnlljjh&g$#dbb}v{^]\ZvYtsVqTohmPO +NchgIe^G\EC_^]?U>SX;VU8S6QPI210..,,*FED&<%:#!!}}{{yywwu3t10qoo,+*#(ihge#d!~a|_zyxqvYtWVTTinm +OkdMbKIedFbaDY^]\>T=R;997SR4PI2G0..,HG)E>'~~;:981x5vus1r/.-n+*)(!hg$e"!b``{zyxwpuXsVqTR +nmPkjihg`eHcFa`B^W at UZ=XW99TSRQPOH1L/J-++ at E(CBA$?>=<;:3zy654t,s*/p-,m*#jih}$#"b~av{^yx[voXWVT +TRRPPNNLhgfH^G\ECCA]\[=S!~;|98y6w4-ts0q.-n+l)"i&gfd"!~`v_t]r[pYWs +VqpoRmlejMLKIIGGEECCAA??==;W:88MRQP2H1FK.IH+F)DCB;$?"!6;|zz16w43ss0/.-&+l)j'hff{"c~}|_zyxwpu +XWVTpSQQfOdMbgfeG]F[DBB@@>><<::88664P311FKJI+A*?D'BA$?"=<;:92yx54t2+r).o,+l)"i&gfd"!~`|_ty\w +vuXmrUTShQOOMMKKIIGGEECCAA?[ZY;Q:OT7RQ4O2MF/.-B+ at EDC%;$9>!<;|9z76/4uts*qoommkki'&f${dyb`|{]y +r[pYWsVqpSnQlkdiLKJ_dcbDZCXA??==;;99775QPO1G0E.,,*F)''=<;49zyxvvttrrp.-,l$k"iggeeccaa +__]][wvuWmVkpSnmPkNihgfe^GbEDYBW\[ZZ=XW:U8SRQPOHM0/.,,**((&&$@?>~6}4{yyw5vtt+0/.n&m +$kiig%$#cybw|_]]rwZutWrUponmlkdMhgJeHcEEZ_^AV?Z=XW9UN7LQ422GL/JIHA*E('%%##!!}}{987w/v-trrppn +nlljjh&g$#ccx}`{^yxZvoXmrUponQfkNMLaJHHFba`BXAV?==;WVU7M6K4220L/--BGFE'=&;@#>=~;|92y6wvttrrp +.-,l*k"'h%$#d!~w|_^yxZvoXmrUSShmPkjMhKfed]FaD_^]?U>S<::8866442200..,H+FE''BA@?8=~;|{y7x543t1 +0/.',m*kjh&geez!b}|{t]\wZXXmrqpRhQfOdMKgfeG]F[DBB@\[Z=<;:3 +z7x5vtt+0q.-,m*)('&%|e"!b}`{]]rwvYnWVUSSQQOOMihJf_H]bE`_B]V?Z='%%##!= +<|:3z1xvvttr0q.-mm*)(!h%f#"!aw`uz]xwvYtsrkTSRPPNNLhgfH^G\ECCAA??==;;997S6QP311LKJIBG*)DCB$:# +8=~;:{8y6543,1rqp'nl*)i'~g|ec!~}_u^s\ZZXXVVTTRRPPNNLhKfeHcFa`_^]V?>YYXW9O8M6442200.JIH*@) +>'%A$?>!}}:98705vutr0q.-n+l)('&}$e"cbw`^zyxZpYnWUUSonmOeNcLJJHdcEaZCXA??==;W:UT66QPONMF/J-HG +F(>'=~;:9870w4ut+0/o-&m$kiiggeeccaa__]yxwYoXmrUponQlkjihg`eHcFECCX]@>>SX;VUT7RQPONMFK.I +,+ at E(&&;@?>~6}49zxx/4u210)p-n+ljj!&g$#"yb}`_t][wvuWmVkTRRPPNNLLJJHHFFDDB^A\[Z=R;:97SRQ3I2G0. +.,,**((&&;$">=};4{2yw5v32s0q.',m*k('g%|ezcaa__]][[YYWWUUSSQmPNNchKfeGGbaZC^A\[Z'%%:?>=}5|3zx654t,s*qo-n+*k(i&%${"cb}|{]s\qZXXmrqSohQfkNihKfIdcb[`CBAV?=YXW9O8MR5PO +200KJIHA*)(=B%@?"=~;:981xwvt210p(o&+l)(i&}f#dcaa__]][[pYWWUqpRngPeNLLJfedF\EZCAA??==R;9U8SRQ +4IN10/--++)E(CB%##>7~}|3876v.u,sqqoommkkiig%$#cybw|_zy\wZunWVUSoRPPejiKg`I^GEECCAA??=YXW9O8M +64P3NM0..IHAF)('%A$?>!<}:927x5vussqqoom+*j(!h}fddb~}|^t]r[pYnWUqTonPPkjibKfIdcbDZCX]@[ZY7<}|{yywwu321q)p'nlljjhhf$#"bxav_]][wZXXmrqpRhQfOMMKgJedGbE`_^]\U>=X; +VU86L5P32GL/JIHA*E('<%:#8=~;:9z16wvus10p.-n%*)i'~g|ecca}`^^sxwYunWlUSoRmlOjMhaJIHFba`BXAV?== +;WV8TM6K4220LKJ,B+@)''%%##!!};|98yww43,1rq.omm$)j'&g$e"!x}`{^yxwYoXmVTTRRPlkjLbK`IGcbaCYBW@> +><<::88664P3NM//JIHA*E(C&$$9>!<;:{876/vutrrppn,m*)jhh%$#"y~a`_tyxwYoXmVTTRnmlNdMbKIIGGEECCA] +@[Z=X;VUTSLQ4O21//-IHG)?(=&$@?>~6}4{yywwuussqqo-n+*jj'&%$#zc~a`^z]xwvYtsrqpiRmPkjiKaJ_HFF[DY +B@\[=YR;PU8SRQJ321/K.,,AFED&<%:#!!}}{{yywwuussq/p-,mkk"'hgfd"c~}`{^sx[ZYWsrqSiRgPNNLLJJHHFFD +`_^@V?T=;;9U8SR5P3NMLKJC,+*((&&$$""~~5:9y70w.3t10q.o,+*)('~%fe"caav_ty\ZZotWrqToRmlkjihafIdG +F[DBB@@U>!<;:{8765432+rqpnnl*)(h~g|eccaa__]][[YutsUkTinQlkNibKJe +HFF[`C^]@[T=X;:88M64PON0F/D-++))''%%##!!};|98xx/4u2s0qoo&+l)('h}$edcx}|^zy\qvYtsVqTohQlOjMKK +`eHcbDD_^W\?Z=X;99NS6QPO2MLEJ-,G*((=BA#?8!6;|98y6w432+r/p-nll#(i&%ee"!~}v{^y\wZXXmrUponQlkji +bgJIdGbaD_B]\[ZYR;VU8S6Q332MLKJIB+F)D'%%:?"=<||987654-2s0q.omm$)j'&%f#"!~}|uz]x[vYWWlqpoQgPe +NLLJJHHFFDDBB@@>>=};4{2ywwuussqqoommk)('g}f{"c~}`{^s\wZYWsrTpiRg +lOjiKKf_dGbE`CAAV[>YXW:UNS65P311FKJ,HA*?D'BA$?"=<5|9z7xvv-2s0/oo,+*#(i&g$eccx}`{zy\wvunsVUpS +QQfkjiKaJ_dGbaD_B]\[ZSY<<<::8866442200.JIH*@)>C&A@?"=<;:3z7xwuussqqoommkk"iggeec!b``uzy +xZpYnsVqpoRmlkjibgJIH]FDDY^]\>T=R;9977553311//--++)EDC%;$9>!<;|9z765432+rq.o,+l)j'&%$#"!x}`_ +^s\ZZXXVVkTRRPlkMibK`IGGEECCAA??=Y<::OT7RQ4O2MLKJIHGF?(CB%@#>~~5:9z1x5vussqqoommk)('g}f{dbb` +`^^\\ZvYWWlqTonmfONiLgfIdG\aDC^A\[>Y7<}|{yywwuussqqoom+ljj!h}$e +"!b}`{zyxwpYtWrUSShmPNNchKfedGba`_^]V[>=<:VUT6L5J311//-IHG)?(=&$$""~~||z8yww.321q)p',m*)j'h% +$#"!~}v_z]x[YYnsVTTinQlkjMhgfedcbaZ_BA at U>S'%%:?>=}5|3zxxv4uss*/.n,%l#jh +hffddbb``^^\x[YYnsVqpoRgPONLLJfedF\EZCAA?[ZC&%@#!!6;|98y6w4-2sr/.-m%l# +jhhffddbb``^^\\ZZXXmrUSShmlkMcLafIdcFaD_^W@?><<::88664422G0..,,*F)''!}}49z76w4-ts0q +oo&+l)(i&g|#d!ba__]][[YYWWUUSSQQOkNLLafedF\EZ_B]\[>YR;:U8SR5P3NMFK.I,+@)''%%:?>=}5|3zxxvvttr +rppnnllj('&f|ez!b}|{^yxwpYtWVkTinQlkjcLKfIGG\aDBBW\?ZY'%%##!!}}{{yywwu3 +21q)p',m*)(i&}fe"caav{^\\qvYtsVqTonglOjMLJJHHFFDDB^]?[T=R;997755331MLK-C,A*(D'BA@#>=<5|{8yww +.3trr).o,+l)j'&%${"cba__]yxwYoXmVTponPfOdMKKIIGGEECCA]@[Z=;;VUTSRQJO210.J-HG*E(CBA@?>7<}:{zx +xv43s1*q(om+*)i!h}fddbb``^^\\ZZXXVrUponQlkjihgf_HGFD`_A]V?T=;;9UTS5K4I200..,,**((&&$@#>=~;4{ +z7xvv-2s0/p-&m*kjhhffddbb``^^s\ZZXXVrUSShmlkMcLafIdcbEZ_BA\[Z><<:V9TS6Q4ONMF/J-H+))>C&$$9>! +<;:{8765.3tsr)pnn%*)(h~g|#dbbw|_zy\wZutsrqjSnQlOMMbgJHH]bE`_^A\[ZYXWPU87644220LKJ,B+@)''<%## +!!}}{{yy05v32s0)p-nmkki'&f${dyb`|{]yx[putVrkTiRPlkMibK`IGGEECCAA??=Y=XWV8N7L53ONM/K.CHGF( +>'!<}:98765.u2s0qoo&+ljj!&g$#"c~}|{zyxqvYtWrqpRhQfkNihg`IdGF[DBBW\[=YR;P977553311//--+G +FE'=&;@#>=<}4{8yxvvttrrppn,+*j"i~ge#"!aw`u^s\qZXtWrqpSnglONMKKIIGGEECCAA??==;WV8TM6KP311FK.I +H+F)DC<%$?"=<}:{876/4ut10/o'n%ljjhhffddbb``^^sx[vuXsVqpongPkNiLJJ_dGEEZ_B]\[>YXWVUNS65P311FK +.,,AF)DC&A$?>=<;:3z7x5vtt+0qoo&+l)('h%$#"!~}v{^y\[YYnWUUSonPleNcLJJHdGba`YB]@[ZY;Q:O866K4IN1 +LKJC,+F)''<YX; +V9TSL5P3N1//DI,**?D'BA@#>=<5:{8yxvv-trrp.-m+$k"igge#d!~``{zyxwpuXsVqpoQgPejMhgfIdcba`Y^A@[ZY +;Q:O86R533HML.JC,A*((&&$$""~~||zzx654t,s*/p-,m*k('&%$#zc~a`^^\\ZvuWslUjSQQOOMMKKIIGGEaDBBW\? +ZYX;VUTSRQPIN10/D-++))>C&A@#>!<;:98765.u2srp.-,l$k"iggeeccaa__]][[YutsUkTinQlkjcLgJI^cbD`YBW +@>Z=XWV9NS6Q4311F/--+GF(D=&;$">=<|4{2ywwuussqqo-n+*)j'~g$e"!a}v_t][[YutsUkTiRPPNNLLJJHH]FDDB +B@\?ZYX;VUNS6Q4311FKJI+A*?(&&$$""~~||zzxxvvt2sqq(-n+*)j'&%|ed!b}|_z]xwvunsVqTSQQOOMMKgJHH]ba +C_^AV[Z7~}:{87x5v3210/.',ml)('g}f{dbb``^^\\ZZXXVVTponPfOdiLgfIdG +ba`_^]\U>Y7<}:9z7x5uu,10q(onm$ki'&%e{dyb``^^\\ZvutVlUjSQQOOMMKKIe +HFF[`C^]@[T=X;VUT6L5J31M0..C,AFED&<%:?"~~5:{876w.utsqq(ommkkii~%$#cybw`^^\\ZZXXVVTpSnmPkNibg +JeHGEECCAA?[Z'=~;:3zy6wuu,1rpp',m*)j'h%$#z!b}`{zy[qZoXVrqpRhQfOMM +KKIIGGEECCAA??=Y><<::O866442N1//DIHG)?(=B%@?>!6;|9zyww.3trr).-,l$k"ig%$#cybw`^^\\ZZXXVVTTR +RPlOMMbgJedcFaZCBAV?TY!<;:{87654-ts0qoo&+ljj!&g$#d!b}|{zyxq +vYtWVkTRRPPNNcLafedF\EZCAA??==;;997755331M0KJIB+*EDC%;$9"~<}{{2y0543s+r).omm$)j'&g$ezc~a`^^\ +\ZZXtsrTjShQOOMMKKIIGGEaD_^]@[TY<;:8866442NM/KD-B+))''%%##!!};:z81x/4u21r/p-,%l)j'hff{"caav{ +^yxwZutslqToRQOOdihgI_H]FDDBB@@>><<::88664P311FK.IHG*EDCB;$#>!<;|9z76543,1rq.o,+lj"i&g$#"b~a +v{^yxwpYXWUqpoQgPeNLLJJHHFFDDBB@@>>~<5|3zxxvvttrrp.-,l$k"'h%$e"c +~w|_^][[putsUkTiRPlkjLbK`IGcbD`YBW@>><<::8866442N1//DI,GF)D'BA:#"!}}{{yy0wu32r0)p'nlljjhhffd +"!~`v_ty\wvYtWrqpinQPkjiKaJ_HFbECCXAV[ZY;Q:OT755JO2ML/J-HGFE>'&%##!!}}{{yyww.ussqqo-nll#('&f +|ez!b}|_z]xwvutmrUTSQQOkjLhaJ_HFbaC_^AV[Z!<}:98765.ut10p.'n%*kii +~g|#d!~a|_zyxwvutmrUpoRmPkMMbgfI^GFEC_^]?U>S<::8866442200..,H+FE(C<%@#"7~5:98x0w.us10/o'n%lj +jh&%e#zcxa__]][[YYWWUqpoQgPejMhgfI^GFEC_B@@UZ=XW:U8SLQ4O21F/--BG*((=BA@"8!6;|987x54-ts0qoo&+ +ljj!&g$#d!b}|{ty\wZYWWUUSSQQOOMMKKIIGcbD`YBW\?==RW:UTS6QPONG0/J-HG*E(CBA@?8=~;|987w5v-2s0/.' +nmljj!&%$dzcxa_{zy[qZoXVrqSohQfOMMKKIIGGEECCA]\[=S~6}4{yyw543s+r) +pnnlljjhhffd"c~}|_zsx[ZuXVVkpSQQfOdMbgJedGbE`_XA\?Z=;;PU866KP3NML/JIHAF)D'&$$">=<|4{2ywwu321 +q)p'nlljjhhff{"c~}|_zyxwpYtWVTTRRPPNNLLJJ_dcbDZCXA??==;;99775Q4ONM0KJIHG at E('BA@"8!6}{987w/v- +trrppnnlljjhhffddb~a|{^y\wvutsrkToRQOkNLLaJ_H]ba`BXAV?==;;9977553311//-I,GFE>'B%$""~~||z876v +.u,sqqoommkkiig%f#"!bw|_z]\ZZoXVrqpRhQfOMMKKIIGGEECCAA?[><=<| +4{2yw543s+r)pnnlljjhhf$e"!~w`_z][[puXVVkTinQlkNLLafIHcbD`YBW\?ZY7 +~;|{yywwuussqq(-,+k#j!hffddbb``^^\x[vutWrqjoRmPOMihgI_H]FDDB^]\>T=R;9977553311//--BG*EDC&A@? +8!~}4{2ywwuus10p.'n%ljjhhffddbb``^zyxZpYnsVqpSnQlkjibgJIdcbDZCXA?[><=6}|{yywwuussqqoo&mkkiig%fddy~}|^t]rwZutWrUponmlkdiLgJedFb[DYB@\[Z!}}49z76w4u21*/ponlljjh&geez!~}_u^s\ZvutVlUjSQQOOMMKKIeHcbECC^]\[TY<;:8T7RQ4O2MLK +JCH+FE(C&A##8=<}4{8y65u3,10/.'nml#jhhff{"!~`v_t][[YutsUkTiRPPNNLLJJHHFFD`C^]@[>S<;V977LQ4ON1 +//JCHGF)D&&;@#>=~;|92765vussq/.-m%l#jhhf$#"bxav_]y\ZZoXW2qjiR.-e=)KgJ%^]F!~C}W@[ZY;WPbTSqK#m +2k}ih,gTF)bPO%:"K7I54zW7gvv-sr*N.'JI[6FE&fUeAR>P+u:9[[pYW3lkS/.QyON*bKJ%dcF!m_^W@>-<;W:sN6%4 +]n[MjEWz,GFd'&s`#L]~6;|WW7UBeuc1qNpLJIk6FEgD1{zyQ=|*:([775WVrUSoAQ,Od*KJJ%HFF!!}}|?.Z=;QPtTq +%4o31kj/WIyfSRbC<`MLo\<|k{2V0fv-Qb=q.o&JH#G4~V$Bdy>P_;](x8vH5"3UpSh.fe=ib(J%7cF!`2B{i.Z+{]]88Y6XslT0B.zl,=<;(J%d]F!`}BW at yyY+d +tO8Mq5PINkjih-BTecQCa`qp>J~5XzW165eR,bO/L^m8[6j'D%UBdc>}`N^9x&vonF2qCSRmf>M*;J&8^]\n~}}@?[xY ++:Pt8S6o]3l~Y..,,*@RQ Added: pypy/dist/pypy/lang/malbolge/examples/copy.mbs ============================================================================== --- (empty file) +++ pypy/dist/pypy/lang/malbolge/examples/copy.mbs Sat Jul 12 11:58:35 2008 @@ -0,0 +1,12 @@ +#!/usr/bin/env malbolge +# --relaxed +# +# copy, written by Lou Scheffer +# +# copies input to output +# +# This application depends on a bug in the original Malbolge +# interpreter. It may stop functioning, if the bug is fixed +# and may not work when run on other interpreters. + +D'BA@?>=<;:9876543210/.-,+*)('&%$#"!~}|{zyxwvutsrqponmlkjihgfedcba`_^]\[ZYXWVUTSRQPONMLKJIHGFEDCBA@?>=<;:9876543210/.-,+*)('&%$#"!~}|{zyxwvutsrqponmlkjihgfedcba`_^]\[ZYXWVUTSRQPONMLKJIHGFEDC&_???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? Added: pypy/dist/pypy/lang/malbolge/examples/hello-world.mb ============================================================================== --- (empty file) +++ pypy/dist/pypy/lang/malbolge/examples/hello-world.mb Sat Jul 12 11:58:35 2008 @@ -0,0 +1 @@ +(=<`$9]7<5YXz7wT.3,+O/o'K%$H"'~D|#z at b=`{^Lx8%$Xmrkpohm-kNi;gsedcba`_^]\[ZYXWVUTSRQPONMLKJIHGFEDCBA@?>=<;:9876543s+O=<;:9876543s+OqxdRp0wkrUo[D7,XTcA"lI''' + \ + r'''.v%{gJh4G\-=O at 5`_3iU!pJS72FhOA1C''' + \ + r'''B6v^=I_0/8|jsb9m<.TVac`uY*MK'X~xDl}REokN:#?G"i@''' ] + self.cipher_length = len(self.decode_cipher) + self.WORD_SIZE = 10 + self.REG_MAX = self.tri2dec([2,2,2,2,2,2,2,2,2,2]) + self.MEM_SIZE = self.REG_MAX + 1 + self.instructions = ['j', 'i', '*', 'p', '<', '/', 'v', 'o'] + self.real_io_operations = book + self.output = RStringIO() + + if source: + self.load( source, relax ) + self.run() + + print self.output.getvalue() + + def error(self, msg): + print "ERROR: " + msg + raise MBException() + + def get_mem(self, ofs): + u_ofs = ofs + assert u_ofs >= 0 + return self.mem[u_ofs] + + def set_mem(self, ofs, value): + u_ofs = ofs + assert u_ofs >= 0 + self.mem[u_ofs] = value + + def is_graphical_ascii(self, c): + return 33 <= c <= 126 + + def dec2tri(self, d): + i = self.WORD_SIZE + t = [0,0,0,0,0,0,0,0,0,0] + while d > 0: + i -= 1 + t[i] = d % 3 + d = d / 3 + return t + + def tri2dec(self, t): + d = 0 + i = 0 + while i < self.WORD_SIZE-1: + d = (d + t[i]) * 3 + i += 1 + return d + t[i] + + def op(self, at, dt): + table = [ + [1,0,0], + [1,0,2], + [2,2,1] + ] + o = [0,0,0,0,0,0,0,0,0,0] + for i in xrange(self.WORD_SIZE): + o[i] = table[dt[i]][at[i]] + return o + + def validate_source(self, length): + pass + # if [c for c in self.mem[(:length)] if not self.is_graphical_ascii(c)]: + # self.error('source code contains invalid character(s)') + # valid = set(self.instructions) + # used = set([self.decode(_) for _ in xrange(length)]) + # if len(used - valid) > 0: + # self.error('source code contains invalid instruction(s)') + + def load(self, source, relax=False): + if len(source) < 2: + self.error('source code too short: must be no shorter than 2') + if len(source) > self.MEM_SIZE: + self.error('source code too long: must be no longer than ' + str(self.MEM_SIZE)) + code = [ord(_) for _ in source if not _.isspace()] + self.mem = code + for i in range(self.MEM_SIZE - len(code)): + self.mem.append(0) + if not relax: + self.validate_source(len(code)) + for i in xrange(len(code),self.MEM_SIZE): + at = self.dec2tri(self.mem[i-1]) + dt = self.dec2tri(self.mem[i-2]) + self.mem[i] = self.tri2dec(self.op(at, dt)) + + def i_setdata(self): + self.d = self.get_mem(self.d) + + def i_setcode(self): + self.c = self.get_mem(self.d) + + def i_rotate(self): + t = self.dec2tri( self.get_mem(self.d)) + size = len(t) - 1 + assert size >= 0 + t = t[size:] + t[:size] + self.set_mem(self.d, self.tri2dec(t)) + self.a = self.get_mem(self.d) + + def i_op(self): + at = self.dec2tri(self.a) + dt = self.dec2tri(self.get_mem(self.d)) + self.set_mem(self.d, self.tri2dec(self.op(at, dt))) + self.a = self.get_mem(self.d) + + def i_write(self): + self.output.write(chr(self.a & 0xff)) + + def i_read(self): + raise NotImplementedError + x = self.input.read(1) + if (len(x) < 1): #EOF + self.a = self.REG_MAX + else: + self.a = ord(x) + + def i_stop(self): + pass + + def i_nop(self): + pass + + def validate(self, addr): + if not self.is_graphical_ascii(self.get_mem(self.c)): + self.error('illegal code detected') + + def decode(self,addr): + return self.decode_cipher[( self.get_mem(addr) - 33 + addr ) % self.cipher_length] + + def is_running(self): + return self.i != 'v' + + def fetch(self): + self.validate(self.get_mem(self.c)) + self.i = self.decode(self.c) + + def execute(self): + if self.i == 'j': self.i_setdata() + elif self.i == 'i': self.i_setcode() + elif self.i == '*': self.i_rotate() + elif self.i == 'p': self.i_op() + elif self.i == 'v': self.i_stop() + elif self.i == 'o': self.i_nop() + else: + if self.real_io_operations: + if self.i == '<': self.i_read() + elif self.i == '/': self.i_write() + else: + if self.i == '<': self.i_write() + elif self.i == '/': self.i_read() + + def modify(self): + self.set_mem(self.c, self.encode_cipher[self.get_mem(self.c) - 33]) + + def increment_c(self): + self.c += 1 + if self.c > self.REG_MAX: + self.c = 0 + + def increment_d(self): + self.d += 1 + if self.d > self.REG_MAX: + self.d = 0 + + def run(self): + self.a = 0 # accumulator + self.c = 0 # code pointer + self.d = 0 # data pointer + self.i = '' # instruction + while self.is_running(): + self.fetch() + self.execute() + self.modify() + self.increment_c() + self.increment_d() + +def entry_point(args): + f = open_file_as_stream(args[1]) + source = f.readall() + f.close() + + relax = '--relaxed' in args + book = '--by-the-book' in args + + try: + Vm(source,relax=relax,book=book) + except MBException, e: + return 1 + except KeyboardInterrupt: + pass + return 0 + +def target(driver, args): + return entry_point, None From arigo at codespeak.net Sat Jul 12 13:00:21 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 13:00:21 +0200 (CEST) Subject: [pypy-svn] r56485 - pypy/branch/always-oldstyle Message-ID: <20080712110021.3194616A00C@codespeak.net> Author: arigo Date: Sat Jul 12 13:00:19 2008 New Revision: 56485 Added: pypy/branch/always-oldstyle/ - copied from r56484, pypy/dist/ Log: (fijal, arigo) A branch in which we remove the --oldstyle option. From arigo at codespeak.net Sat Jul 12 13:01:24 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 13:01:24 +0200 (CEST) Subject: [pypy-svn] r56486 - pypy/branch/always-oldstyle/demo/sqlinj Message-ID: <20080712110124.75A9016A00C@codespeak.net> Author: arigo Date: Sat Jul 12 13:01:23 2008 New Revision: 56486 Removed: pypy/branch/always-oldstyle/demo/sqlinj/ Log: Kill old broken demo. From arigo at codespeak.net Sat Jul 12 13:02:44 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 13:02:44 +0200 (CEST) Subject: [pypy-svn] r56487 - in pypy/branch/always-oldstyle: lib-python lib-python/modified-2.4.1 lib-python/modified-2.4.1/test pypy pypy/config pypy/doc/config pypy/interpreter pypy/interpreter/test pypy/module/__builtin__ pypy/module/__builtin__/test pypy/module/operator/test pypy/objspace/std pypy/objspace/std/test pypy/objspace/test pypy/translator/goal Message-ID: <20080712110244.62A9D16A00F@codespeak.net> Author: arigo Date: Sat Jul 12 13:02:42 2008 New Revision: 56487 Removed: pypy/branch/always-oldstyle/lib-python/modified-2.4.1/types.py pypy/branch/always-oldstyle/pypy/doc/config/objspace.std.oldstyle.txt Modified: pypy/branch/always-oldstyle/lib-python/conftest.py pypy/branch/always-oldstyle/lib-python/modified-2.4.1/test/test_repr.py pypy/branch/always-oldstyle/pypy/config/pypyoption.py pypy/branch/always-oldstyle/pypy/conftest.py pypy/branch/always-oldstyle/pypy/interpreter/baseobjspace.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_appinterp.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_class.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_compiler.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_exec.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_function.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_interpreter.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_objspace.py pypy/branch/always-oldstyle/pypy/interpreter/test/test_raise.py pypy/branch/always-oldstyle/pypy/module/__builtin__/__init__.py pypy/branch/always-oldstyle/pypy/module/__builtin__/operation.py pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_builtin.py pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_classobj.py pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_descriptor.py pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_functional.py pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_reduce.py pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_vars.py pypy/branch/always-oldstyle/pypy/module/operator/test/test_operator.py pypy/branch/always-oldstyle/pypy/objspace/std/objspace.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_complexobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_dictobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_floatobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_index.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_intobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_iterobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_function.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_internals.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_stringobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_typeobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_unicodeobject.py pypy/branch/always-oldstyle/pypy/objspace/std/test/test_userobject.py pypy/branch/always-oldstyle/pypy/objspace/test/test_descriptor.py pypy/branch/always-oldstyle/pypy/translator/goal/app_main.py Log: Remove the --oldstyle option. Fix a few small bugs in the tests. Port hopefully all tests that used "class A:" to create new-style classes. Modified: pypy/branch/always-oldstyle/lib-python/conftest.py ============================================================================== --- pypy/branch/always-oldstyle/lib-python/conftest.py (original) +++ pypy/branch/always-oldstyle/lib-python/conftest.py Sat Jul 12 13:02:42 2008 @@ -297,24 +297,16 @@ class RegrTest: """ Regression Test Declaration.""" def __init__(self, basename, enabled=False, dumbtest=False, - oldstyle=False, core=False, + core=False, compiler=None, usemodules = ''): self.basename = basename self.enabled = enabled self.dumbtest = dumbtest - # we have to determine the value of oldstyle - # lazily because at RegrTest() call time the command - # line options haven't been parsed! - self._oldstyle = oldstyle self._usemodules = usemodules.split() self._compiler = compiler self.core = core - def oldstyle(self): - return self._oldstyle #or pypy_option.oldstyle - oldstyle = property(oldstyle) - def usemodules(self): return self._usemodules #+ pypy_option.usemodules usemodules = property(usemodules) @@ -325,7 +317,7 @@ def getoptions(self): l = [] - for name in 'oldstyle', 'core': + for name in ['core']: if getattr(self, name): l.append(name) for name in self.usemodules: @@ -365,18 +357,13 @@ self._prepare(space) fspath = self.getfspath() assert fspath.check() - if self.oldstyle: - space.enable_old_style_classes_as_default_metaclass() - try: - modname = fspath.purebasename - space.appexec([], '''(): - from test import %(modname)s - m = %(modname)s - if hasattr(m, 'test_main'): - m.test_main() - ''' % locals()) - finally: - space.enable_new_style_classes_as_default_metaclass() + modname = fspath.purebasename + space.appexec([], '''(): + from test import %(modname)s + m = %(modname)s + if hasattr(m, 'test_main'): + m.test_main() + ''' % locals()) testmap = [ RegrTest('test___all__.py', enabled=True, core=True), @@ -422,7 +409,7 @@ RegrTest('test_cgi.py', enabled=True), RegrTest('test_charmapcodec.py', enabled=True, core=True), RegrTest('test_cl.py', enabled=False, dumbtest=1), - RegrTest('test_class.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_class.py', enabled=True, core=True), RegrTest('test_cmath.py', enabled=True, dumbtest=1, core=True), RegrTest('test_codeccallbacks.py', enabled=True, core=True), RegrTest('test_codecencodings_cn.py', enabled=False), @@ -438,11 +425,11 @@ RegrTest('test_codecmaps_tw.py', enabled=False), RegrTest('test_codecs.py', enabled=True, core=True), RegrTest('test_codeop.py', enabled=True, core=True), - RegrTest('test_coercion.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_coercion.py', enabled=True, core=True), RegrTest('test_colorsys.py', enabled=True), RegrTest('test_commands.py', enabled=True), - RegrTest('test_compare.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_compare.py', enabled=True, core=True), RegrTest('test_compile.py', enabled=True, core=True), RegrTest('test_compiler.py', enabled=True, core=False), # this test tests the compiler package from stdlib RegrTest('test_complex.py', enabled=True, core=True), @@ -463,8 +450,8 @@ RegrTest('test_decimal.py', enabled=True), RegrTest('test_decorators.py', enabled=True, core=True), RegrTest('test_deque.py', enabled=True, core=True), - RegrTest('test_descr.py', enabled=True, core=True, oldstyle=True, usemodules='_weakref'), - RegrTest('test_descrtut.py', enabled=True, core=True, oldstyle=True), + RegrTest('test_descr.py', enabled=True, core=True, usemodules='_weakref'), + RegrTest('test_descrtut.py', enabled=True, core=True), RegrTest('test_dict.py', enabled=True, core=True), RegrTest('test_difflib.py', enabled=True, dumbtest=1), @@ -577,7 +564,7 @@ RegrTest('test_multifile.py', enabled=True), RegrTest('test_mutants.py', enabled=True, dumbtest=1, core="possibly"), RegrTest('test_netrc.py', enabled=True), - RegrTest('test_new.py', enabled=True, core=True, oldstyle=True), + RegrTest('test_new.py', enabled=True, core=True), RegrTest('test_nis.py', enabled=False), RegrTest('test_normalization.py', enabled=False), RegrTest('test_ntpath.py', enabled=True, dumbtest=1), @@ -635,7 +622,7 @@ RegrTest('test_re.py', enabled=True, core=True), RegrTest('test_regex.py', enabled=False), - RegrTest('test_repr.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_repr.py', enabled=True, core=True), #rev 10840: 6 of 12 tests fail. Always minor stuff like #'' != '' @@ -903,11 +890,6 @@ if option.use_compiled: execpath, info = getexecutable() pypy_options = [] - if regrtest.oldstyle: - if (option.use_compiled and - not info.get('objspace.std.oldstyle', False)): - py.test.skip("old-style classes not available with this pypy-c") - pypy_options.append('--oldstyle') if regrtest.compiler: pypy_options.append('--compiler=%s' % regrtest.compiler) pypy_options.extend( Modified: pypy/branch/always-oldstyle/lib-python/modified-2.4.1/test/test_repr.py ============================================================================== --- pypy/branch/always-oldstyle/lib-python/modified-2.4.1/test/test_repr.py (original) +++ pypy/branch/always-oldstyle/lib-python/modified-2.4.1/test/test_repr.py Sat Jul 12 13:02:42 2008 @@ -97,9 +97,6 @@ eq(r(n), expected) def test_instance(self): - # Disabled for PyPy because it relies on oldstyle class behaviour. - # Running the test under oldstyle results in many more other problems - # though. eq = self.assertEquals i1 = ClassWithRepr("a") eq(r(i1), repr(i1)) @@ -134,8 +131,9 @@ # Functions eq(repr(hash), '') # Methods - self.failUnless(repr(''.split).find( - "bound method str.split of '' at 0x") > -1) + self.failUnless('method' in repr(''.split)) + self.failUnless('str' in repr(''.split)) + self.failUnless('split' in repr(''.split)) def test_xrange(self): import warnings @@ -174,7 +172,9 @@ def test_descriptors(self): eq = self.assertEquals # method descriptors - eq(repr(dict.items), "") + self.assert_('method' in repr(dict.items)) + self.assert_('dict' in repr(dict.items)) + self.assert_('items' in repr(dict.items)) # XXX member descriptors # XXX attribute descriptors # XXX slot descriptors @@ -222,8 +222,7 @@ os.remove(p) del sys.path[0] - def DONOTtest_module(self): - # PyPy really doesn't (want to) do these complex module reprs. + def test_module(self): eq = self.assertEquals touch(os.path.join(self.subpkgname, self.pkgname + os.extsep + 'py')) from areallylongpackageandmodulenametotestreprtruncation.areallylongpackageandmodulenametotestreprtruncation import areallylongpackageandmodulenametotestreprtruncation Modified: pypy/branch/always-oldstyle/pypy/config/pypyoption.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/config/pypyoption.py (original) +++ pypy/branch/always-oldstyle/pypy/config/pypyoption.py Sat Jul 12 13:02:42 2008 @@ -287,10 +287,6 @@ "track types that override __getattribute__", default=False), - BoolOption("oldstyle", - "specify whether the default metaclass should be classobj", - default=False, cmdline="-k --oldstyle"), - BoolOption("logspaceoptypes", "a instrumentation option: before exit, print the types seen by " "certain simpler bytecodes", Modified: pypy/branch/always-oldstyle/pypy/conftest.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/conftest.py (original) +++ pypy/branch/always-oldstyle/pypy/conftest.py Sat Jul 12 13:02:42 2008 @@ -90,8 +90,6 @@ "module %r required" % (modname,)) continue if info is None: - if key == 'objspace.std.oldstyle' and value: - continue # fine on CPython py.test.skip("cannot runappdirect this test on top of CPython") has = info.get(key, None) if has != value: Modified: pypy/branch/always-oldstyle/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/baseobjspace.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/baseobjspace.py Sat Jul 12 13:02:42 2008 @@ -207,7 +207,7 @@ full_exceptions = True # full support for exceptions (normalization & more) - def __init__(self, config=None, **kw): + def __init__(self, config=None): "NOT_RPYTHON: Basic initialization of objects." self.fromcache = InternalSpaceCache(self).getorbuild self.threadlocals = ThreadLocals() @@ -227,7 +227,6 @@ self.frame_trace_action = FrameTraceAction(self) self.actionflag.register_action(self.user_del_action) self.actionflag.register_action(self.frame_trace_action) - self.setoptions(**kw) from pypy.interpreter.pyframe import PyFrame self.FrameClass = PyFrame # can be overridden to a subclass @@ -237,10 +236,6 @@ self.initialize() - def setoptions(self): - # override this in subclasses for extra-options - pass - def startup(self): # To be called before using the space @@ -735,10 +730,15 @@ return w_value return None + def is_oldstyle_instance(self, w_obj): + # xxx hack hack hack + from pypy.module.__builtin__.interp_classobj import W_InstanceObject + obj = self.interpclass_w(w_obj) + return obj is not None and isinstance(obj, W_InstanceObject) + def callable(self, w_obj): if self.lookup(w_obj, "__call__") is not None: - w_is_oldstyle = self.isinstance(w_obj, self.w_instance) - if self.is_true(w_is_oldstyle): + if self.is_oldstyle_instance(w_obj): # ugly old style class special treatment, but well ... try: self.getattr(w_obj, self.wrap("__call__")) Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_appinterp.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_appinterp.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_appinterp.py Sat Jul 12 13:02:42 2008 @@ -86,7 +86,7 @@ def test_applevel_class(space, applevel_temp = applevel_temp): app = applevel_temp(''' - class C: + class C(object): clsattr = 42 def __init__(self, x=13): self.attr = x Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_class.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_class.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_class.py Sat Jul 12 13:02:42 2008 @@ -46,28 +46,28 @@ assert c.__class__ == C def test_method(self): - class C: + class C(object): def meth(self): return 1 c = C() assert c.meth() == 1 def test_method_exc(self): - class C: + class C(object): def meth(self): raise RuntimeError c = C() raises(RuntimeError, c.meth) def test_class_attr(self): - class C: + class C(object): a = 42 c = C() assert c.a == 42 assert C.a == 42 def test_class_attr_inherited(self): - class C: + class C(object): a = 42 class D(C): pass @@ -110,7 +110,7 @@ assert float(x) == 5.5 def test_meth_doc(self): - class C: + class C(object): def meth_no_doc(self): pass def meth_doc(self): Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_compiler.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_compiler.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_compiler.py Sat Jul 12 13:02:42 2008 @@ -350,7 +350,7 @@ def test_mangling(self): snippet = str(py.code.Source(r''' __g = "42" - class X: + class X(object): def __init__(self, u): self.__u = u def __f(__self, __n): @@ -390,7 +390,7 @@ def test_chained_access_augassign(self): snippet = str(py.code.Source(r''' - class R: + class R(object): count = 0 c = 0 for i in [0,1,2]: @@ -480,7 +480,7 @@ def f(self): def get_nested_class(): self - class Test: + class Test(object): def _STOP_HERE_(self): return _STOP_HERE_(self) get_nested_class() @@ -496,7 +496,7 @@ space = self.space snippet = str(py.code.Source(r''' def f(x): - class Test: + class Test(object): def meth(self): return x + 1 return Test() @@ -533,7 +533,7 @@ snippet = str(py.code.Source(r''' def f(): method_and_var = "var" - class Test: + class Test(object): def method_and_var(self): return "method" def test(self): @@ -630,7 +630,7 @@ class AppTestOptimizer: def test_constant_fold_add(self): import parser - class Folder: + class Folder(object): def defaultvisit(self, node): return node Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_exec.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_exec.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_exec.py Sat Jul 12 13:02:42 2008 @@ -119,7 +119,7 @@ def test_nested_names_are_not_confused(self): def get_nested_class(): method_and_var = "var" - class Test: + class Test(object): def method_and_var(self): return "method" def test(self): Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_function.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_function.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_function.py Sat Jul 12 13:02:42 2008 @@ -191,7 +191,7 @@ def test_get_get(self): # sanxiyn's test from email def m(self): return self - class C: pass + class C(object): pass class D(C): pass C.m = m D.m = C.m @@ -201,7 +201,7 @@ assert d.m() == d def test_method_eq(self): - class C: + class C(object): def m(): pass c = C() assert C.m == C.m @@ -229,10 +229,6 @@ assert repr(A.f) == "" assert repr(A().f).startswith("" @@ -240,13 +236,13 @@ def test_method_call(self): - class C: + class C(object): def __init__(self, **kw): pass c = C(type='test') def test_method_w_callable(self): - class A: + class A(object): def __call__(self, x): return x import new @@ -254,7 +250,7 @@ assert im() == 3 def test_method_w_callable_call_function(self): - class A: + class A(object): def __call__(self, x, y): return x+y import new Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_interpreter.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_interpreter.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_interpreter.py Sat Jul 12 13:02:42 2008 @@ -263,7 +263,7 @@ def test_print(self): import sys save = sys.stdout - class Out: + class Out(object): def __init__(self): self.args = [] def write(self, *args): Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_objspace.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_objspace.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_objspace.py Sat Jul 12 13:02:42 2008 @@ -108,7 +108,7 @@ w_oldstyle = self.space.appexec([], """(): class NoCall: - __metaclass__ = _classobj + pass return NoCall()""") assert not is_callable(w_oldstyle) self.space.setattr(w_oldstyle, self.space.wrap("__call__"), w_func) Modified: pypy/branch/always-oldstyle/pypy/interpreter/test/test_raise.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/interpreter/test/test_raise.py (original) +++ pypy/branch/always-oldstyle/pypy/interpreter/test/test_raise.py Sat Jul 12 13:02:42 2008 @@ -96,7 +96,7 @@ raises(StopIteration, f) def test_userclass(self): - class A: + class A(object): def __init__(self, x=None): self.x = x class B(A): @@ -124,7 +124,8 @@ assert b.x == 42 def test_it(self): - C = _classobj('C', (), {}) + class C: + pass # this used to explode in the exception normalization step: try: {}[C] Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/__init__.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/__init__.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/__init__.py Sat Jul 12 13:02:42 2008 @@ -68,11 +68,8 @@ 'file' : 'state.get(space).w_file', 'open' : 'state.get(space).w_file', - # old-style classes dummy support - '_classobj' : 'interp_classobj.W_ClassObject', - '_instance' : 'interp_classobj.W_InstanceObject', - # default __metaclass__ - '__metaclass__' : '(space.w_type)', + # default __metaclass__: old-style class + '__metaclass__' : 'interp_classobj.W_ClassObject', # interp-level function definitions 'abs' : 'operation.abs', Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/operation.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/operation.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/operation.py Sat Jul 12 13:02:42 2008 @@ -181,20 +181,6 @@ """ if w_sentinel is None: return space.iter(w_collection_or_callable) - # XXX it seems that CPython checks the following - # for newstyle but doesn't for oldstyle classes :-( - #w_res = space.iter(w_collection_or_callable) - #w_typeres = space.type(w_res) - #try: - # space.getattr(w_typeres, space.wrap('next')) - #except OperationError, e: - # if not e.match(space, space.w_AttributeError): - # raise - # raise OperationError(space.w_TypeError, - # space.wrap("iter() returned non-iterator of type '%s'" % - # w_typeres.name)) - #else: - # return w_res else: return iter_sentinel(space, w_collection_or_callable, w_sentinel) Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_builtin.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_builtin.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_builtin.py Sat Jul 12 13:02:42 2008 @@ -75,16 +75,16 @@ def nosp(x): return [y for y in x if y[0]!='_'] assert f() == [] assert g() == ['a', 'b', 'c'] - class X: pass + class X(object): pass assert nosp(dir(X)) == [] - class X: + class X(object): a = 23 c = 45 b = 67 assert nosp(dir(X)) == ['a', 'b', 'c'] def test_dir_in_broken_locals(self): - class C: + class C(object): def __getitem__(self, item): raise KeyError(item) def keys(self): @@ -100,7 +100,7 @@ assert g() == {'a':0, 'b':0, 'c':0} def test_getattr(self): - class a: + class a(object): i = 5 assert getattr(a, 'i') == 5 raises(AttributeError, getattr, a, 'k') @@ -282,7 +282,7 @@ def test_reversed_custom_objects(self): """make sure __reversed__ is called when defined""" - class SomeClass: + class SomeClass(object): def __reversed__(self): return 42 obj = SomeClass() @@ -297,7 +297,7 @@ assert cmp(u"abc", 12) != 0 def test_cmp_more(self): - class C: + class C(object): def __eq__(self, other): return True def __cmp__(self, other): @@ -332,7 +332,7 @@ raises(TypeError,coerce, u'a' , 'a') def test_return_None(self): - class X: pass + class X(object): pass x = X() assert setattr(x, 'x', 11) == None assert delattr(x, 'x') == None @@ -343,7 +343,7 @@ assert divmod(15,10) ==(1,5) def test_callable(self): - class Call: + class Call(object): def __call__(self, a): return a+2 assert callable(Call()), ( @@ -351,10 +351,6 @@ assert callable(int), ( "Builtin function 'callable' misreads int") class Call: - try: - __metaclass__ = _classobj - except NameError: # not running on PyPy, assuming oldstyle implicitely - pass def __call__(self, a): return a+2 assert callable(Call()) @@ -374,10 +370,7 @@ assert not callable(a), ( "Builtin function 'callable' tricked by instance-__call__") class NoCall: - try: - __metaclass__ = _classobj - except NameError: # not running on PyPy, assuming oldstyle implicitely - pass + pass assert not callable(NoCall()) def test_hash(self): @@ -435,7 +428,7 @@ raises(TypeError, issubclass, int, (float, 6)) def test_staticmethod(self): - class X: + class X(object): def f(*args, **kwds): return args, kwds f = staticmethod(f) assert X.f() == ((), {}) @@ -444,7 +437,7 @@ assert X().f(42, x=43) == ((42,), {'x': 43}) def test_classmethod(self): - class X: + class X(object): def f(*args, **kwds): return args, kwds f = classmethod(f) class Y(X): Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_classobj.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_classobj.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_classobj.py Sat Jul 12 13:02:42 2008 @@ -1,8 +1,5 @@ class AppTestOldstyle(object): - def setup_class(cls): - from pypy.conftest import gettestobjspace - cls.space = gettestobjspace(**{"objspace.std.oldstyle": True}) def test_simple(self): class A: Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_descriptor.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_descriptor.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_descriptor.py Sat Jul 12 13:02:42 2008 @@ -3,7 +3,7 @@ class AppTestBuiltinApp: def test_staticmethod(self): - class C: + class C(object): def f(a, b): return a+b f = staticmethod(f) @@ -18,7 +18,7 @@ assert D.f("abc", "def") == "abcdef" def test_classmethod(self): - class C: + class C(object): def f(cls, stuff): return cls, stuff f = classmethod(f) @@ -189,7 +189,7 @@ assert C.__dict__["f"](p) == "B.f->C.f" def test_super_errors(self): - class C: + class C(object): pass class D(C): pass @@ -231,7 +231,7 @@ def test_property_docstring(self): assert property.__doc__.startswith('property') - class A: + class A(object): pass A.x = property(lambda x: x, lambda x, y: x, lambda x:x, 'xxx') Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_functional.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_functional.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_functional.py Sat Jul 12 13:02:42 2008 @@ -156,10 +156,10 @@ def test_all(self): - class TestFailingBool: + class TestFailingBool(object): def __nonzero__(self): raise RuntimeError - class TestFailingIter: + class TestFailingIter(object): def __iter__(self): raise RuntimeError @@ -178,10 +178,10 @@ def test_any(self): - class TestFailingBool: + class TestFailingBool(object): def __nonzero__(self): raise RuntimeError - class TestFailingIter: + class TestFailingIter(object): def __iter__(self): raise RuntimeError Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_reduce.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_reduce.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_reduce.py Sat Jul 12 13:02:42 2008 @@ -14,7 +14,7 @@ assert reduce(lambda x, y: x-y, [2, 8], 10) == 0 def test_from_cpython(self): - class SequenceClass: + class SequenceClass(object): def __init__(self, n): self.n = n def __getitem__(self, i): Modified: pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_vars.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_vars.py (original) +++ pypy/branch/always-oldstyle/pypy/module/__builtin__/test/test_vars.py Sat Jul 12 13:02:42 2008 @@ -9,7 +9,7 @@ raises(TypeError, vars, 0, 1) def test_vars_correct_arguments(self): - class a: + class a(object): def __init__(self): self.res = 42 assert vars(a) == a.__dict__ Modified: pypy/branch/always-oldstyle/pypy/module/operator/test/test_operator.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/module/operator/test/test_operator.py (original) +++ pypy/branch/always-oldstyle/pypy/module/operator/test/test_operator.py Sat Jul 12 13:02:42 2008 @@ -6,7 +6,7 @@ def test_getters_are_not_regular_functions(self): import operator - class A: + class A(object): getx = operator.attrgetter('x') get3 = operator.itemgetter(3) a = A() Modified: pypy/branch/always-oldstyle/pypy/objspace/std/objspace.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/objspace.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/objspace.py Sat Jul 12 13:02:42 2008 @@ -63,10 +63,6 @@ PACKAGE_PATH = 'objspace.std' - def setoptions(self, **kwds): - if "oldstyle" in kwds: - self.config.objspace.std.oldstyle = kwds["oldstyle"] - def initialize(self): "NOT_RPYTHON: only for initializing the space." self._typecache = Cache() @@ -314,10 +310,8 @@ self.make_builtins() self.sys.setmodule(w_mod) - # dummy old-style classes types - self.w_classobj = W_TypeObject(self, 'classobj', [self.w_object], {}) - self.w_instance = W_TypeObject(self, 'instance', [self.w_object], {}) - self.setup_old_style_classes() + # the type of old-style classes + self.w_classobj = self.builtin.get('__metaclass__') # fix up a problem where multimethods apparently don't # like to define this at interp-level @@ -336,9 +330,6 @@ """) self.w_dict.__flags__ = old_flags - if self.config.objspace.std.oldstyle: - self.enable_old_style_classes_as_default_metaclass() - # final setup self.setup_builtin_modules() # Adding transparent proxy call @@ -351,23 +342,6 @@ self.setattr(w___pypy__, self.wrap('get_tproxy_controller'), self.wrap(app_proxy_controller)) - def enable_old_style_classes_as_default_metaclass(self): - self.setitem(self.builtin.w_dict, self.wrap('__metaclass__'), self.w_classobj) - - def enable_new_style_classes_as_default_metaclass(self): - space = self - try: - self.delitem(self.builtin.w_dict, self.wrap('__metaclass__')) - except OperationError, e: - if not e.match(space, space.w_KeyError): - raise - - def setup_old_style_classes(self): - """NOT_RPYTHON""" - # sanity check that this approach is working and is not too late - self.w_classobj = self.getattr(self.builtin, self.wrap('_classobj')) - self.w_instance = self.getattr(self.builtin, self.wrap('_instance')) - def create_builtin_module(self, pyname, publicname): """NOT_RPYTHON helper function which returns the wrapped module and its dict. Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_complexobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_complexobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_complexobject.py Sat Jul 12 13:02:42 2008 @@ -59,6 +59,8 @@ class AppTestAppComplexTest: def setup_class(cls): + # XXX these tests probably copied directly from CPython + # please port them to pypy style :-/ cls.w_helper = cls.space.appexec([], """ (): import sys Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_dictobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_dictobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_dictobject.py Sat Jul 12 13:02:42 2008 @@ -314,7 +314,7 @@ assert str(d) == '{0: {...}}' # Mutating while repr'ing - class Machiavelli: + class Machiavelli(object): def __repr__(self): d.clear() return "42" Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_floatobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_floatobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_floatobject.py Sat Jul 12 13:02:42 2008 @@ -91,7 +91,7 @@ assert 0.0 == round(22.22222, -2) def test_special_float_method(self): - class a: + class a(object): def __float__(self): self.ar = True return None @@ -99,7 +99,7 @@ raises(TypeError, float, inst) assert inst.ar - class b: + class b(object): pass raises((AttributeError, TypeError), float, b()) Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_index.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_index.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_index.py Sat Jul 12 13:02:42 2008 @@ -3,7 +3,7 @@ class AppTest_IndexProtocol: def setup_class(self): - self.space = gettestobjspace(oldstyle=True) + self.space = gettestobjspace() w_oldstyle = self.space.appexec([], """(): class oldstyle: def __index__(self): Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_intobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_intobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_intobject.py Sat Jul 12 13:02:42 2008 @@ -386,7 +386,7 @@ assert (j(100) >> 2, type(j(100) >> 2)) == ( 25, int) def test_special_int(self): - class a: + class a(object): def __int__(self): self.ar = True return None @@ -394,12 +394,12 @@ raises(TypeError, int, inst) assert inst.ar == True - class b: + class b(object): pass raises((AttributeError,TypeError), int, b()) def test_special_long(self): - class a: + class a(object): def __long__(self): self.ar = True return None @@ -407,7 +407,7 @@ raises(TypeError, long, inst) assert inst.ar == True - class b: + class b(object): pass raises((AttributeError,TypeError), long, b()) Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_iterobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_iterobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_iterobject.py Sat Jul 12 13:02:42 2008 @@ -38,7 +38,7 @@ class AppTestW_IterObjectApp: def test_user_iter(self): - class C: + class C(object): def next(self): raise StopIteration def __iter__(self): @@ -46,13 +46,13 @@ assert list(C()) == [] def test_iter_getitem(self): - class C: + class C(object): def __getitem__(self, i): return range(2)[i] assert list(C()) == range(2) def test_iter_fail_noseq(self): - class C: + class C(object): pass raises(TypeError, iter, Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_function.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_function.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_function.py Sat Jul 12 13:02:42 2008 @@ -50,7 +50,7 @@ def f(self): return 3 - class A: + class A(object): pass fun = self.get_proxy(f) @@ -81,18 +81,18 @@ assert f.__doc__ is None def test_proxy_bind_method(self): - class A: + class A(object): pass def f(self): return 3 - class AA: + class AA(object): pass from __pypy__ import tproxy as proxy a = A() - class X: + class X(object): def __init__(self, x): self.x = x def f(self, name, *args, **kwargs): Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_internals.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_internals.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_proxy_internals.py Sat Jul 12 13:02:42 2008 @@ -129,29 +129,3 @@ pass t = tproxy(file, f) - -class DONTAppTestProxyType(AppProxy): - def test_filetype(self): - f = self.get_proxy(file) - f("/tmp/sth", "w").write("aaa") - assert open("/tmp/sth").read() == "aaa" - - def test_fileobject(self): - f = open("/tmp/sth", "w") - fp = self.get_proxy(f) - fp.write("aaa") - fp.close() - assert open("/tmp/sth").read() == "aaa" - - def test_isinstance(self): - class A: - pass - - a = A() - Ap = self.get_proxy(A) - ap = self.get_proxy(a) - assert isinstance(a, A) - assert isinstance(a, Ap) - assert isinstance(ap, A) - assert isinstance(ap, Ap) - assert type(a) is type(ap) Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_stringobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_stringobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_stringobject.py Sat Jul 12 13:02:42 2008 @@ -509,7 +509,7 @@ def test_unicode_join_endcase(self): # This class inserts a Unicode object into its argument's natural # iteration, in the 3rd position. - class OhPhooey: + class OhPhooey(object): def __init__(self, seq): self.it = iter(seq) self.i = 0 Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_typeobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_typeobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_typeobject.py Sat Jul 12 13:02:42 2008 @@ -215,15 +215,9 @@ assert 0, "shouldn't be able to create inheritance cycles" # let's throw a classic class into the mix: - try: - class Classic: - __metaclass__ = _classobj - def meth2(self): - return 3 - except NameError: - class Classic: - def meth2(self): - return 3 + class Classic: + def meth2(self): + return 3 D.__bases__ = (C, Classic) @@ -369,27 +363,14 @@ assert type(HasInnerMetaclass) == HasInnerMetaclass.__metaclass__ def test_implicit_metaclass(self): - global __metaclass__ - try: - old_metaclass = __metaclass__ - has_old_metaclass = True - except NameError: - has_old_metaclass = False - class __metaclass__(type): pass - class HasImplicitMetaclass: - pass - - try: - assert type(HasImplicitMetaclass) == __metaclass__ - finally: - if has_old_metaclass: - __metaclass__ = old_metaclass - else: - del __metaclass__ + g = {'__metaclass__': __metaclass__} + exec "class HasImplicitMetaclass: pass\n" in g + HasImplicitMetaclass = g['HasImplicitMetaclass'] + assert type(HasImplicitMetaclass) == __metaclass__ def test_mro(self): class A_mro(object): @@ -410,8 +391,8 @@ assert getattr(B_mro(), 'a', None) == None def test_abstract_mro(self): - class A1: - __metaclass__ = _classobj + class A1: # old-style class + pass class B1(A1): pass class C1(A1): @@ -595,7 +576,7 @@ def test_only_classic_bases_fails(self): class C: - __metaclass__ = _classobj + pass raises(TypeError, type, 'D', (C,), {}) def test_set___class__(self): Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_unicodeobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_unicodeobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_unicodeobject.py Sat Jul 12 13:02:42 2008 @@ -269,17 +269,10 @@ assert U(u'test').__class__ is U def test_call_unicode(self): - skip("does not work") - class X: + class X(object): def __unicode__(self): return u'x' - - try: - unicode(X(), 'ascii') - except TypeError, t: - assert 'need string or buffer' in str(t) - else: - raise Exception("DID NOT RAISE") + raises(TypeError, unicode, X(), 'ascii') def test_startswith(self): assert u'ab'.startswith(u'ab') is True Modified: pypy/branch/always-oldstyle/pypy/objspace/std/test/test_userobject.py ============================================================================== --- pypy/branch/always-oldstyle/pypy/objspace/std/test/test_userobject.py (original) +++ pypy/branch/always-oldstyle/pypy/objspace/std/test/test_userobject.py Sat Jul 12 13:02:42 2008 @@ -8,21 +8,21 @@ cls.space = conftest.gettestobjspace(**cls.OPTIONS) def test_emptyclass(self): - class empty: pass + class empty(object): pass inst = empty() assert isinstance(inst, empty) - inst.attr=23 - assert inst.attr ==23 + inst.attr = 23 + assert inst.attr == 23 def test_method(self): - class A: + class A(object): def f(self, v): return v*42 a = A() assert a.f('?') == '??????????????????????????????????????????' def test_unboundmethod(self): - class A: + class A(object): def f(self, v): return v*17 a = A() @@ -46,7 +46,7 @@ assert len(result) ==3 def test_subsubclass(self): - class base: + class base(object): baseattr = 12 class derived(base): derivedattr = 34 @@ -56,7 +56,7 @@ assert inst.derivedattr ==34 def test_descr_get(self): - class C: + class C(object): class desc(object): def __get__(self, ob, cls=None): return 42 @@ -86,21 +86,21 @@ assert c.wibble == 22 def test_class_setattr(self): - class C: + class C(object): pass C.a = 1 assert hasattr(C, 'a') assert C.a == 1 def test_add(self): - class C: + class C(object): def __add__(self, other): return self, other c1 = C() assert c1+3 == (c1, 3) def test_call(self): - class C: + class C(object): def __call__(self, *args): return args c1 = C() @@ -118,7 +118,7 @@ assert c1.a == '->a' def test_getattr(self): - class C: + class C(object): def __getattr__(self, name): return '->' + name c1 = C() @@ -207,12 +207,6 @@ s = repr(Foo()) assert s.startswith('> sys.stderr, msg @@ -221,7 +216,6 @@ i = 0 run_module = False run_stdin = False - oldstyle_classes = False unbuffered = False while i < len(argv): arg = argv[i] @@ -260,8 +254,6 @@ return 2 run_module = True break - elif arg in ('-k', '--oldstyle'): - oldstyle_classes = True elif arg == '--': i += 1 break # terminates option list @@ -287,10 +279,6 @@ mainmodule = type(sys)('__main__') sys.modules['__main__'] = mainmodule - if oldstyle_classes: - import __builtin__ - __builtin__.__metaclass__ = __builtin__._classobj - if import_site: try: import site From hpk at codespeak.net Sat Jul 12 14:46:51 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sat, 12 Jul 2008 14:46:51 +0200 (CEST) Subject: [pypy-svn] r56488 - pypy/extradoc/planning/1.1 Message-ID: <20080712124651.E021B2A8084@codespeak.net> Author: hpk Date: Sat Jul 12 14:46:49 2008 New Revision: 56488 Added: pypy/extradoc/planning/1.1/ pypy/extradoc/planning/1.1/plan.txt (contents, props changed) Log: a first list of todo's and goals for a 1.1 release, after discussions with anto and samuele. Added: pypy/extradoc/planning/1.1/plan.txt ============================================================================== --- (empty file) +++ pypy/extradoc/planning/1.1/plan.txt Sat Jul 12 14:46:49 2008 @@ -0,0 +1,33 @@ +1.1 release goals: PyPy Python Interpreter + - compliant to CPython 2.5 + - ctypes + - robustness + - source distribution that installs and + runs on Linux, OSX, Windows + - supports easy_install / setuptools + - works with django, pylons and many other apps + - features stackless, transparent proxies, xxx sandboxing + - experimental .NET and JVM versions?? + +major todos: + +- cleanup of directories and maybe modularizing +- documentation about writing new extension modules +- integrate SoC 2.5 work, port lib-python to 2.5, fix remaining 2.5 issues +- review ctypes skipped tests +- threading works nicely +- a build process that integrates well with distutils, + easy_install should fail gracefully for c-extensions +- setup automated windows testing, integrate testing "pypy-c" + on all platforms, **have one central pypy-c testing page** +- check missing standard library modules +- trunk/dist separation, dist is automatically copied from + trunk after all tests have passed +- refactor documentation to have separated entry points: + - for users of the mainline PyPy Python Interpreter + - for advanced users/developers/experimenters of the PyPy Python + Interpreter + - all the rest - toolchain, prolog, squeak, javascript +- revise and cleanup play1 - run more things through sandboxing? +- debian packaging? +- check licensing/copyright situation and boilerplate From arigo at codespeak.net Sat Jul 12 14:55:46 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 14:55:46 +0200 (CEST) Subject: [pypy-svn] r56489 - in pypy/dist: demo/sqlinj lib-python lib-python/modified-2.4.1 lib-python/modified-2.4.1/test pypy pypy/config pypy/doc/config pypy/interpreter pypy/interpreter/test pypy/module/__builtin__ pypy/module/__builtin__/test pypy/module/operator/test pypy/objspace/std pypy/objspace/std/test pypy/objspace/test pypy/translator/goal Message-ID: <20080712125546.3C5C92A8084@codespeak.net> Author: arigo Date: Sat Jul 12 14:55:43 2008 New Revision: 56489 Removed: pypy/dist/demo/sqlinj/ pypy/dist/lib-python/modified-2.4.1/types.py pypy/dist/pypy/doc/config/objspace.std.oldstyle.txt Modified: pypy/dist/lib-python/conftest.py pypy/dist/lib-python/modified-2.4.1/test/test_repr.py pypy/dist/pypy/config/pypyoption.py pypy/dist/pypy/conftest.py pypy/dist/pypy/interpreter/baseobjspace.py pypy/dist/pypy/interpreter/test/test_appinterp.py pypy/dist/pypy/interpreter/test/test_class.py pypy/dist/pypy/interpreter/test/test_compiler.py pypy/dist/pypy/interpreter/test/test_exec.py pypy/dist/pypy/interpreter/test/test_function.py pypy/dist/pypy/interpreter/test/test_interpreter.py pypy/dist/pypy/interpreter/test/test_objspace.py pypy/dist/pypy/interpreter/test/test_raise.py pypy/dist/pypy/module/__builtin__/__init__.py pypy/dist/pypy/module/__builtin__/operation.py pypy/dist/pypy/module/__builtin__/test/test_builtin.py pypy/dist/pypy/module/__builtin__/test/test_classobj.py pypy/dist/pypy/module/__builtin__/test/test_descriptor.py pypy/dist/pypy/module/__builtin__/test/test_functional.py pypy/dist/pypy/module/__builtin__/test/test_reduce.py pypy/dist/pypy/module/__builtin__/test/test_vars.py pypy/dist/pypy/module/operator/test/test_operator.py pypy/dist/pypy/objspace/std/objspace.py pypy/dist/pypy/objspace/std/test/test_complexobject.py pypy/dist/pypy/objspace/std/test/test_dictobject.py pypy/dist/pypy/objspace/std/test/test_floatobject.py pypy/dist/pypy/objspace/std/test/test_index.py pypy/dist/pypy/objspace/std/test/test_intobject.py pypy/dist/pypy/objspace/std/test/test_iterobject.py pypy/dist/pypy/objspace/std/test/test_proxy_function.py pypy/dist/pypy/objspace/std/test/test_proxy_internals.py pypy/dist/pypy/objspace/std/test/test_stringobject.py pypy/dist/pypy/objspace/std/test/test_typeobject.py pypy/dist/pypy/objspace/std/test/test_unicodeobject.py pypy/dist/pypy/objspace/std/test/test_userobject.py pypy/dist/pypy/objspace/test/test_descriptor.py pypy/dist/pypy/translator/goal/app_main.py Log: (fijal, arigo) Merge the always-oldstyle branch: Remove the --oldstyle option. Fix a few small bugs in the tests. Port hopefully all tests that used "class A:" to create new-style classes. Kill old broken demo. Modified: pypy/dist/lib-python/conftest.py ============================================================================== --- pypy/dist/lib-python/conftest.py (original) +++ pypy/dist/lib-python/conftest.py Sat Jul 12 14:55:43 2008 @@ -297,24 +297,16 @@ class RegrTest: """ Regression Test Declaration.""" def __init__(self, basename, enabled=False, dumbtest=False, - oldstyle=False, core=False, + core=False, compiler=None, usemodules = ''): self.basename = basename self.enabled = enabled self.dumbtest = dumbtest - # we have to determine the value of oldstyle - # lazily because at RegrTest() call time the command - # line options haven't been parsed! - self._oldstyle = oldstyle self._usemodules = usemodules.split() self._compiler = compiler self.core = core - def oldstyle(self): - return self._oldstyle #or pypy_option.oldstyle - oldstyle = property(oldstyle) - def usemodules(self): return self._usemodules #+ pypy_option.usemodules usemodules = property(usemodules) @@ -325,7 +317,7 @@ def getoptions(self): l = [] - for name in 'oldstyle', 'core': + for name in ['core']: if getattr(self, name): l.append(name) for name in self.usemodules: @@ -365,18 +357,13 @@ self._prepare(space) fspath = self.getfspath() assert fspath.check() - if self.oldstyle: - space.enable_old_style_classes_as_default_metaclass() - try: - modname = fspath.purebasename - space.appexec([], '''(): - from test import %(modname)s - m = %(modname)s - if hasattr(m, 'test_main'): - m.test_main() - ''' % locals()) - finally: - space.enable_new_style_classes_as_default_metaclass() + modname = fspath.purebasename + space.appexec([], '''(): + from test import %(modname)s + m = %(modname)s + if hasattr(m, 'test_main'): + m.test_main() + ''' % locals()) testmap = [ RegrTest('test___all__.py', enabled=True, core=True), @@ -422,7 +409,7 @@ RegrTest('test_cgi.py', enabled=True), RegrTest('test_charmapcodec.py', enabled=True, core=True), RegrTest('test_cl.py', enabled=False, dumbtest=1), - RegrTest('test_class.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_class.py', enabled=True, core=True), RegrTest('test_cmath.py', enabled=True, dumbtest=1, core=True), RegrTest('test_codeccallbacks.py', enabled=True, core=True), RegrTest('test_codecencodings_cn.py', enabled=False), @@ -438,11 +425,11 @@ RegrTest('test_codecmaps_tw.py', enabled=False), RegrTest('test_codecs.py', enabled=True, core=True), RegrTest('test_codeop.py', enabled=True, core=True), - RegrTest('test_coercion.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_coercion.py', enabled=True, core=True), RegrTest('test_colorsys.py', enabled=True), RegrTest('test_commands.py', enabled=True), - RegrTest('test_compare.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_compare.py', enabled=True, core=True), RegrTest('test_compile.py', enabled=True, core=True), RegrTest('test_compiler.py', enabled=True, core=False), # this test tests the compiler package from stdlib RegrTest('test_complex.py', enabled=True, core=True), @@ -463,8 +450,8 @@ RegrTest('test_decimal.py', enabled=True), RegrTest('test_decorators.py', enabled=True, core=True), RegrTest('test_deque.py', enabled=True, core=True), - RegrTest('test_descr.py', enabled=True, core=True, oldstyle=True, usemodules='_weakref'), - RegrTest('test_descrtut.py', enabled=True, core=True, oldstyle=True), + RegrTest('test_descr.py', enabled=True, core=True, usemodules='_weakref'), + RegrTest('test_descrtut.py', enabled=True, core=True), RegrTest('test_dict.py', enabled=True, core=True), RegrTest('test_difflib.py', enabled=True, dumbtest=1), @@ -577,7 +564,7 @@ RegrTest('test_multifile.py', enabled=True), RegrTest('test_mutants.py', enabled=True, dumbtest=1, core="possibly"), RegrTest('test_netrc.py', enabled=True), - RegrTest('test_new.py', enabled=True, core=True, oldstyle=True), + RegrTest('test_new.py', enabled=True, core=True), RegrTest('test_nis.py', enabled=False), RegrTest('test_normalization.py', enabled=False), RegrTest('test_ntpath.py', enabled=True, dumbtest=1), @@ -635,7 +622,7 @@ RegrTest('test_re.py', enabled=True, core=True), RegrTest('test_regex.py', enabled=False), - RegrTest('test_repr.py', enabled=True, oldstyle=True, core=True), + RegrTest('test_repr.py', enabled=True, core=True), #rev 10840: 6 of 12 tests fail. Always minor stuff like #'' != '' @@ -903,11 +890,6 @@ if option.use_compiled: execpath, info = getexecutable() pypy_options = [] - if regrtest.oldstyle: - if (option.use_compiled and - not info.get('objspace.std.oldstyle', False)): - py.test.skip("old-style classes not available with this pypy-c") - pypy_options.append('--oldstyle') if regrtest.compiler: pypy_options.append('--compiler=%s' % regrtest.compiler) pypy_options.extend( Modified: pypy/dist/lib-python/modified-2.4.1/test/test_repr.py ============================================================================== --- pypy/dist/lib-python/modified-2.4.1/test/test_repr.py (original) +++ pypy/dist/lib-python/modified-2.4.1/test/test_repr.py Sat Jul 12 14:55:43 2008 @@ -97,9 +97,6 @@ eq(r(n), expected) def test_instance(self): - # Disabled for PyPy because it relies on oldstyle class behaviour. - # Running the test under oldstyle results in many more other problems - # though. eq = self.assertEquals i1 = ClassWithRepr("a") eq(r(i1), repr(i1)) @@ -134,8 +131,9 @@ # Functions eq(repr(hash), '') # Methods - self.failUnless(repr(''.split).find( - "bound method str.split of '' at 0x") > -1) + self.failUnless('method' in repr(''.split)) + self.failUnless('str' in repr(''.split)) + self.failUnless('split' in repr(''.split)) def test_xrange(self): import warnings @@ -174,7 +172,9 @@ def test_descriptors(self): eq = self.assertEquals # method descriptors - eq(repr(dict.items), "") + self.assert_('method' in repr(dict.items)) + self.assert_('dict' in repr(dict.items)) + self.assert_('items' in repr(dict.items)) # XXX member descriptors # XXX attribute descriptors # XXX slot descriptors @@ -222,8 +222,7 @@ os.remove(p) del sys.path[0] - def DONOTtest_module(self): - # PyPy really doesn't (want to) do these complex module reprs. + def test_module(self): eq = self.assertEquals touch(os.path.join(self.subpkgname, self.pkgname + os.extsep + 'py')) from areallylongpackageandmodulenametotestreprtruncation.areallylongpackageandmodulenametotestreprtruncation import areallylongpackageandmodulenametotestreprtruncation Modified: pypy/dist/pypy/config/pypyoption.py ============================================================================== --- pypy/dist/pypy/config/pypyoption.py (original) +++ pypy/dist/pypy/config/pypyoption.py Sat Jul 12 14:55:43 2008 @@ -287,10 +287,6 @@ "track types that override __getattribute__", default=False), - BoolOption("oldstyle", - "specify whether the default metaclass should be classobj", - default=False, cmdline="-k --oldstyle"), - BoolOption("logspaceoptypes", "a instrumentation option: before exit, print the types seen by " "certain simpler bytecodes", Modified: pypy/dist/pypy/conftest.py ============================================================================== --- pypy/dist/pypy/conftest.py (original) +++ pypy/dist/pypy/conftest.py Sat Jul 12 14:55:43 2008 @@ -90,8 +90,6 @@ "module %r required" % (modname,)) continue if info is None: - if key == 'objspace.std.oldstyle' and value: - continue # fine on CPython py.test.skip("cannot runappdirect this test on top of CPython") has = info.get(key, None) if has != value: Modified: pypy/dist/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/dist/pypy/interpreter/baseobjspace.py (original) +++ pypy/dist/pypy/interpreter/baseobjspace.py Sat Jul 12 14:55:43 2008 @@ -207,7 +207,7 @@ full_exceptions = True # full support for exceptions (normalization & more) - def __init__(self, config=None, **kw): + def __init__(self, config=None): "NOT_RPYTHON: Basic initialization of objects." self.fromcache = InternalSpaceCache(self).getorbuild self.threadlocals = ThreadLocals() @@ -227,7 +227,6 @@ self.frame_trace_action = FrameTraceAction(self) self.actionflag.register_action(self.user_del_action) self.actionflag.register_action(self.frame_trace_action) - self.setoptions(**kw) from pypy.interpreter.pyframe import PyFrame self.FrameClass = PyFrame # can be overridden to a subclass @@ -237,10 +236,6 @@ self.initialize() - def setoptions(self): - # override this in subclasses for extra-options - pass - def startup(self): # To be called before using the space @@ -735,10 +730,15 @@ return w_value return None + def is_oldstyle_instance(self, w_obj): + # xxx hack hack hack + from pypy.module.__builtin__.interp_classobj import W_InstanceObject + obj = self.interpclass_w(w_obj) + return obj is not None and isinstance(obj, W_InstanceObject) + def callable(self, w_obj): if self.lookup(w_obj, "__call__") is not None: - w_is_oldstyle = self.isinstance(w_obj, self.w_instance) - if self.is_true(w_is_oldstyle): + if self.is_oldstyle_instance(w_obj): # ugly old style class special treatment, but well ... try: self.getattr(w_obj, self.wrap("__call__")) Modified: pypy/dist/pypy/interpreter/test/test_appinterp.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_appinterp.py (original) +++ pypy/dist/pypy/interpreter/test/test_appinterp.py Sat Jul 12 14:55:43 2008 @@ -86,7 +86,7 @@ def test_applevel_class(space, applevel_temp = applevel_temp): app = applevel_temp(''' - class C: + class C(object): clsattr = 42 def __init__(self, x=13): self.attr = x Modified: pypy/dist/pypy/interpreter/test/test_class.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_class.py (original) +++ pypy/dist/pypy/interpreter/test/test_class.py Sat Jul 12 14:55:43 2008 @@ -46,28 +46,28 @@ assert c.__class__ == C def test_method(self): - class C: + class C(object): def meth(self): return 1 c = C() assert c.meth() == 1 def test_method_exc(self): - class C: + class C(object): def meth(self): raise RuntimeError c = C() raises(RuntimeError, c.meth) def test_class_attr(self): - class C: + class C(object): a = 42 c = C() assert c.a == 42 assert C.a == 42 def test_class_attr_inherited(self): - class C: + class C(object): a = 42 class D(C): pass @@ -110,7 +110,7 @@ assert float(x) == 5.5 def test_meth_doc(self): - class C: + class C(object): def meth_no_doc(self): pass def meth_doc(self): Modified: pypy/dist/pypy/interpreter/test/test_compiler.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_compiler.py (original) +++ pypy/dist/pypy/interpreter/test/test_compiler.py Sat Jul 12 14:55:43 2008 @@ -350,7 +350,7 @@ def test_mangling(self): snippet = str(py.code.Source(r''' __g = "42" - class X: + class X(object): def __init__(self, u): self.__u = u def __f(__self, __n): @@ -390,7 +390,7 @@ def test_chained_access_augassign(self): snippet = str(py.code.Source(r''' - class R: + class R(object): count = 0 c = 0 for i in [0,1,2]: @@ -480,7 +480,7 @@ def f(self): def get_nested_class(): self - class Test: + class Test(object): def _STOP_HERE_(self): return _STOP_HERE_(self) get_nested_class() @@ -496,7 +496,7 @@ space = self.space snippet = str(py.code.Source(r''' def f(x): - class Test: + class Test(object): def meth(self): return x + 1 return Test() @@ -533,7 +533,7 @@ snippet = str(py.code.Source(r''' def f(): method_and_var = "var" - class Test: + class Test(object): def method_and_var(self): return "method" def test(self): @@ -630,7 +630,7 @@ class AppTestOptimizer: def test_constant_fold_add(self): import parser - class Folder: + class Folder(object): def defaultvisit(self, node): return node Modified: pypy/dist/pypy/interpreter/test/test_exec.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_exec.py (original) +++ pypy/dist/pypy/interpreter/test/test_exec.py Sat Jul 12 14:55:43 2008 @@ -119,7 +119,7 @@ def test_nested_names_are_not_confused(self): def get_nested_class(): method_and_var = "var" - class Test: + class Test(object): def method_and_var(self): return "method" def test(self): Modified: pypy/dist/pypy/interpreter/test/test_function.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_function.py (original) +++ pypy/dist/pypy/interpreter/test/test_function.py Sat Jul 12 14:55:43 2008 @@ -191,7 +191,7 @@ def test_get_get(self): # sanxiyn's test from email def m(self): return self - class C: pass + class C(object): pass class D(C): pass C.m = m D.m = C.m @@ -201,7 +201,7 @@ assert d.m() == d def test_method_eq(self): - class C: + class C(object): def m(): pass c = C() assert C.m == C.m @@ -229,10 +229,6 @@ assert repr(A.f) == "" assert repr(A().f).startswith("" @@ -240,13 +236,13 @@ def test_method_call(self): - class C: + class C(object): def __init__(self, **kw): pass c = C(type='test') def test_method_w_callable(self): - class A: + class A(object): def __call__(self, x): return x import new @@ -254,7 +250,7 @@ assert im() == 3 def test_method_w_callable_call_function(self): - class A: + class A(object): def __call__(self, x, y): return x+y import new Modified: pypy/dist/pypy/interpreter/test/test_interpreter.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_interpreter.py (original) +++ pypy/dist/pypy/interpreter/test/test_interpreter.py Sat Jul 12 14:55:43 2008 @@ -263,7 +263,7 @@ def test_print(self): import sys save = sys.stdout - class Out: + class Out(object): def __init__(self): self.args = [] def write(self, *args): Modified: pypy/dist/pypy/interpreter/test/test_objspace.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_objspace.py (original) +++ pypy/dist/pypy/interpreter/test/test_objspace.py Sat Jul 12 14:55:43 2008 @@ -108,7 +108,7 @@ w_oldstyle = self.space.appexec([], """(): class NoCall: - __metaclass__ = _classobj + pass return NoCall()""") assert not is_callable(w_oldstyle) self.space.setattr(w_oldstyle, self.space.wrap("__call__"), w_func) Modified: pypy/dist/pypy/interpreter/test/test_raise.py ============================================================================== --- pypy/dist/pypy/interpreter/test/test_raise.py (original) +++ pypy/dist/pypy/interpreter/test/test_raise.py Sat Jul 12 14:55:43 2008 @@ -96,7 +96,7 @@ raises(StopIteration, f) def test_userclass(self): - class A: + class A(object): def __init__(self, x=None): self.x = x class B(A): @@ -124,7 +124,8 @@ assert b.x == 42 def test_it(self): - C = _classobj('C', (), {}) + class C: + pass # this used to explode in the exception normalization step: try: {}[C] Modified: pypy/dist/pypy/module/__builtin__/__init__.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/__init__.py (original) +++ pypy/dist/pypy/module/__builtin__/__init__.py Sat Jul 12 14:55:43 2008 @@ -68,11 +68,8 @@ 'file' : 'state.get(space).w_file', 'open' : 'state.get(space).w_file', - # old-style classes dummy support - '_classobj' : 'interp_classobj.W_ClassObject', - '_instance' : 'interp_classobj.W_InstanceObject', - # default __metaclass__ - '__metaclass__' : '(space.w_type)', + # default __metaclass__: old-style class + '__metaclass__' : 'interp_classobj.W_ClassObject', # interp-level function definitions 'abs' : 'operation.abs', Modified: pypy/dist/pypy/module/__builtin__/operation.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/operation.py (original) +++ pypy/dist/pypy/module/__builtin__/operation.py Sat Jul 12 14:55:43 2008 @@ -181,20 +181,6 @@ """ if w_sentinel is None: return space.iter(w_collection_or_callable) - # XXX it seems that CPython checks the following - # for newstyle but doesn't for oldstyle classes :-( - #w_res = space.iter(w_collection_or_callable) - #w_typeres = space.type(w_res) - #try: - # space.getattr(w_typeres, space.wrap('next')) - #except OperationError, e: - # if not e.match(space, space.w_AttributeError): - # raise - # raise OperationError(space.w_TypeError, - # space.wrap("iter() returned non-iterator of type '%s'" % - # w_typeres.name)) - #else: - # return w_res else: return iter_sentinel(space, w_collection_or_callable, w_sentinel) Modified: pypy/dist/pypy/module/__builtin__/test/test_builtin.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_builtin.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_builtin.py Sat Jul 12 14:55:43 2008 @@ -75,16 +75,16 @@ def nosp(x): return [y for y in x if y[0]!='_'] assert f() == [] assert g() == ['a', 'b', 'c'] - class X: pass + class X(object): pass assert nosp(dir(X)) == [] - class X: + class X(object): a = 23 c = 45 b = 67 assert nosp(dir(X)) == ['a', 'b', 'c'] def test_dir_in_broken_locals(self): - class C: + class C(object): def __getitem__(self, item): raise KeyError(item) def keys(self): @@ -100,7 +100,7 @@ assert g() == {'a':0, 'b':0, 'c':0} def test_getattr(self): - class a: + class a(object): i = 5 assert getattr(a, 'i') == 5 raises(AttributeError, getattr, a, 'k') @@ -282,7 +282,7 @@ def test_reversed_custom_objects(self): """make sure __reversed__ is called when defined""" - class SomeClass: + class SomeClass(object): def __reversed__(self): return 42 obj = SomeClass() @@ -297,7 +297,7 @@ assert cmp(u"abc", 12) != 0 def test_cmp_more(self): - class C: + class C(object): def __eq__(self, other): return True def __cmp__(self, other): @@ -332,7 +332,7 @@ raises(TypeError,coerce, u'a' , 'a') def test_return_None(self): - class X: pass + class X(object): pass x = X() assert setattr(x, 'x', 11) == None assert delattr(x, 'x') == None @@ -343,7 +343,7 @@ assert divmod(15,10) ==(1,5) def test_callable(self): - class Call: + class Call(object): def __call__(self, a): return a+2 assert callable(Call()), ( @@ -351,10 +351,6 @@ assert callable(int), ( "Builtin function 'callable' misreads int") class Call: - try: - __metaclass__ = _classobj - except NameError: # not running on PyPy, assuming oldstyle implicitely - pass def __call__(self, a): return a+2 assert callable(Call()) @@ -374,10 +370,7 @@ assert not callable(a), ( "Builtin function 'callable' tricked by instance-__call__") class NoCall: - try: - __metaclass__ = _classobj - except NameError: # not running on PyPy, assuming oldstyle implicitely - pass + pass assert not callable(NoCall()) def test_hash(self): @@ -435,7 +428,7 @@ raises(TypeError, issubclass, int, (float, 6)) def test_staticmethod(self): - class X: + class X(object): def f(*args, **kwds): return args, kwds f = staticmethod(f) assert X.f() == ((), {}) @@ -444,7 +437,7 @@ assert X().f(42, x=43) == ((42,), {'x': 43}) def test_classmethod(self): - class X: + class X(object): def f(*args, **kwds): return args, kwds f = classmethod(f) class Y(X): Modified: pypy/dist/pypy/module/__builtin__/test/test_classobj.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_classobj.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_classobj.py Sat Jul 12 14:55:43 2008 @@ -1,8 +1,5 @@ class AppTestOldstyle(object): - def setup_class(cls): - from pypy.conftest import gettestobjspace - cls.space = gettestobjspace(**{"objspace.std.oldstyle": True}) def test_simple(self): class A: Modified: pypy/dist/pypy/module/__builtin__/test/test_descriptor.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_descriptor.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_descriptor.py Sat Jul 12 14:55:43 2008 @@ -3,7 +3,7 @@ class AppTestBuiltinApp: def test_staticmethod(self): - class C: + class C(object): def f(a, b): return a+b f = staticmethod(f) @@ -18,7 +18,7 @@ assert D.f("abc", "def") == "abcdef" def test_classmethod(self): - class C: + class C(object): def f(cls, stuff): return cls, stuff f = classmethod(f) @@ -189,7 +189,7 @@ assert C.__dict__["f"](p) == "B.f->C.f" def test_super_errors(self): - class C: + class C(object): pass class D(C): pass @@ -231,7 +231,7 @@ def test_property_docstring(self): assert property.__doc__.startswith('property') - class A: + class A(object): pass A.x = property(lambda x: x, lambda x, y: x, lambda x:x, 'xxx') Modified: pypy/dist/pypy/module/__builtin__/test/test_functional.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_functional.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_functional.py Sat Jul 12 14:55:43 2008 @@ -156,10 +156,10 @@ def test_all(self): - class TestFailingBool: + class TestFailingBool(object): def __nonzero__(self): raise RuntimeError - class TestFailingIter: + class TestFailingIter(object): def __iter__(self): raise RuntimeError @@ -178,10 +178,10 @@ def test_any(self): - class TestFailingBool: + class TestFailingBool(object): def __nonzero__(self): raise RuntimeError - class TestFailingIter: + class TestFailingIter(object): def __iter__(self): raise RuntimeError Modified: pypy/dist/pypy/module/__builtin__/test/test_reduce.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_reduce.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_reduce.py Sat Jul 12 14:55:43 2008 @@ -14,7 +14,7 @@ assert reduce(lambda x, y: x-y, [2, 8], 10) == 0 def test_from_cpython(self): - class SequenceClass: + class SequenceClass(object): def __init__(self, n): self.n = n def __getitem__(self, i): Modified: pypy/dist/pypy/module/__builtin__/test/test_vars.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_vars.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_vars.py Sat Jul 12 14:55:43 2008 @@ -9,7 +9,7 @@ raises(TypeError, vars, 0, 1) def test_vars_correct_arguments(self): - class a: + class a(object): def __init__(self): self.res = 42 assert vars(a) == a.__dict__ Modified: pypy/dist/pypy/module/operator/test/test_operator.py ============================================================================== --- pypy/dist/pypy/module/operator/test/test_operator.py (original) +++ pypy/dist/pypy/module/operator/test/test_operator.py Sat Jul 12 14:55:43 2008 @@ -6,7 +6,7 @@ def test_getters_are_not_regular_functions(self): import operator - class A: + class A(object): getx = operator.attrgetter('x') get3 = operator.itemgetter(3) a = A() Modified: pypy/dist/pypy/objspace/std/objspace.py ============================================================================== --- pypy/dist/pypy/objspace/std/objspace.py (original) +++ pypy/dist/pypy/objspace/std/objspace.py Sat Jul 12 14:55:43 2008 @@ -63,10 +63,6 @@ PACKAGE_PATH = 'objspace.std' - def setoptions(self, **kwds): - if "oldstyle" in kwds: - self.config.objspace.std.oldstyle = kwds["oldstyle"] - def initialize(self): "NOT_RPYTHON: only for initializing the space." self._typecache = Cache() @@ -314,10 +310,8 @@ self.make_builtins() self.sys.setmodule(w_mod) - # dummy old-style classes types - self.w_classobj = W_TypeObject(self, 'classobj', [self.w_object], {}) - self.w_instance = W_TypeObject(self, 'instance', [self.w_object], {}) - self.setup_old_style_classes() + # the type of old-style classes + self.w_classobj = self.builtin.get('__metaclass__') # fix up a problem where multimethods apparently don't # like to define this at interp-level @@ -336,9 +330,6 @@ """) self.w_dict.__flags__ = old_flags - if self.config.objspace.std.oldstyle: - self.enable_old_style_classes_as_default_metaclass() - # final setup self.setup_builtin_modules() # Adding transparent proxy call @@ -351,23 +342,6 @@ self.setattr(w___pypy__, self.wrap('get_tproxy_controller'), self.wrap(app_proxy_controller)) - def enable_old_style_classes_as_default_metaclass(self): - self.setitem(self.builtin.w_dict, self.wrap('__metaclass__'), self.w_classobj) - - def enable_new_style_classes_as_default_metaclass(self): - space = self - try: - self.delitem(self.builtin.w_dict, self.wrap('__metaclass__')) - except OperationError, e: - if not e.match(space, space.w_KeyError): - raise - - def setup_old_style_classes(self): - """NOT_RPYTHON""" - # sanity check that this approach is working and is not too late - self.w_classobj = self.getattr(self.builtin, self.wrap('_classobj')) - self.w_instance = self.getattr(self.builtin, self.wrap('_instance')) - def create_builtin_module(self, pyname, publicname): """NOT_RPYTHON helper function which returns the wrapped module and its dict. Modified: pypy/dist/pypy/objspace/std/test/test_complexobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_complexobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_complexobject.py Sat Jul 12 14:55:43 2008 @@ -59,6 +59,8 @@ class AppTestAppComplexTest: def setup_class(cls): + # XXX these tests probably copied directly from CPython + # please port them to pypy style :-/ cls.w_helper = cls.space.appexec([], """ (): import sys Modified: pypy/dist/pypy/objspace/std/test/test_dictobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_dictobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_dictobject.py Sat Jul 12 14:55:43 2008 @@ -314,7 +314,7 @@ assert str(d) == '{0: {...}}' # Mutating while repr'ing - class Machiavelli: + class Machiavelli(object): def __repr__(self): d.clear() return "42" Modified: pypy/dist/pypy/objspace/std/test/test_floatobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_floatobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_floatobject.py Sat Jul 12 14:55:43 2008 @@ -91,7 +91,7 @@ assert 0.0 == round(22.22222, -2) def test_special_float_method(self): - class a: + class a(object): def __float__(self): self.ar = True return None @@ -99,7 +99,7 @@ raises(TypeError, float, inst) assert inst.ar - class b: + class b(object): pass raises((AttributeError, TypeError), float, b()) Modified: pypy/dist/pypy/objspace/std/test/test_index.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_index.py (original) +++ pypy/dist/pypy/objspace/std/test/test_index.py Sat Jul 12 14:55:43 2008 @@ -3,7 +3,7 @@ class AppTest_IndexProtocol: def setup_class(self): - self.space = gettestobjspace(oldstyle=True) + self.space = gettestobjspace() w_oldstyle = self.space.appexec([], """(): class oldstyle: def __index__(self): Modified: pypy/dist/pypy/objspace/std/test/test_intobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_intobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_intobject.py Sat Jul 12 14:55:43 2008 @@ -386,7 +386,7 @@ assert (j(100) >> 2, type(j(100) >> 2)) == ( 25, int) def test_special_int(self): - class a: + class a(object): def __int__(self): self.ar = True return None @@ -394,12 +394,12 @@ raises(TypeError, int, inst) assert inst.ar == True - class b: + class b(object): pass raises((AttributeError,TypeError), int, b()) def test_special_long(self): - class a: + class a(object): def __long__(self): self.ar = True return None @@ -407,7 +407,7 @@ raises(TypeError, long, inst) assert inst.ar == True - class b: + class b(object): pass raises((AttributeError,TypeError), long, b()) Modified: pypy/dist/pypy/objspace/std/test/test_iterobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_iterobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_iterobject.py Sat Jul 12 14:55:43 2008 @@ -38,7 +38,7 @@ class AppTestW_IterObjectApp: def test_user_iter(self): - class C: + class C(object): def next(self): raise StopIteration def __iter__(self): @@ -46,13 +46,13 @@ assert list(C()) == [] def test_iter_getitem(self): - class C: + class C(object): def __getitem__(self, i): return range(2)[i] assert list(C()) == range(2) def test_iter_fail_noseq(self): - class C: + class C(object): pass raises(TypeError, iter, Modified: pypy/dist/pypy/objspace/std/test/test_proxy_function.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_proxy_function.py (original) +++ pypy/dist/pypy/objspace/std/test/test_proxy_function.py Sat Jul 12 14:55:43 2008 @@ -50,7 +50,7 @@ def f(self): return 3 - class A: + class A(object): pass fun = self.get_proxy(f) @@ -81,18 +81,18 @@ assert f.__doc__ is None def test_proxy_bind_method(self): - class A: + class A(object): pass def f(self): return 3 - class AA: + class AA(object): pass from __pypy__ import tproxy as proxy a = A() - class X: + class X(object): def __init__(self, x): self.x = x def f(self, name, *args, **kwargs): Modified: pypy/dist/pypy/objspace/std/test/test_proxy_internals.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_proxy_internals.py (original) +++ pypy/dist/pypy/objspace/std/test/test_proxy_internals.py Sat Jul 12 14:55:43 2008 @@ -129,29 +129,3 @@ pass t = tproxy(file, f) - -class DONTAppTestProxyType(AppProxy): - def test_filetype(self): - f = self.get_proxy(file) - f("/tmp/sth", "w").write("aaa") - assert open("/tmp/sth").read() == "aaa" - - def test_fileobject(self): - f = open("/tmp/sth", "w") - fp = self.get_proxy(f) - fp.write("aaa") - fp.close() - assert open("/tmp/sth").read() == "aaa" - - def test_isinstance(self): - class A: - pass - - a = A() - Ap = self.get_proxy(A) - ap = self.get_proxy(a) - assert isinstance(a, A) - assert isinstance(a, Ap) - assert isinstance(ap, A) - assert isinstance(ap, Ap) - assert type(a) is type(ap) Modified: pypy/dist/pypy/objspace/std/test/test_stringobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_stringobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_stringobject.py Sat Jul 12 14:55:43 2008 @@ -509,7 +509,7 @@ def test_unicode_join_endcase(self): # This class inserts a Unicode object into its argument's natural # iteration, in the 3rd position. - class OhPhooey: + class OhPhooey(object): def __init__(self, seq): self.it = iter(seq) self.i = 0 Modified: pypy/dist/pypy/objspace/std/test/test_typeobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_typeobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_typeobject.py Sat Jul 12 14:55:43 2008 @@ -215,15 +215,9 @@ assert 0, "shouldn't be able to create inheritance cycles" # let's throw a classic class into the mix: - try: - class Classic: - __metaclass__ = _classobj - def meth2(self): - return 3 - except NameError: - class Classic: - def meth2(self): - return 3 + class Classic: + def meth2(self): + return 3 D.__bases__ = (C, Classic) @@ -369,27 +363,14 @@ assert type(HasInnerMetaclass) == HasInnerMetaclass.__metaclass__ def test_implicit_metaclass(self): - global __metaclass__ - try: - old_metaclass = __metaclass__ - has_old_metaclass = True - except NameError: - has_old_metaclass = False - class __metaclass__(type): pass - class HasImplicitMetaclass: - pass - - try: - assert type(HasImplicitMetaclass) == __metaclass__ - finally: - if has_old_metaclass: - __metaclass__ = old_metaclass - else: - del __metaclass__ + g = {'__metaclass__': __metaclass__} + exec "class HasImplicitMetaclass: pass\n" in g + HasImplicitMetaclass = g['HasImplicitMetaclass'] + assert type(HasImplicitMetaclass) == __metaclass__ def test_mro(self): class A_mro(object): @@ -410,8 +391,8 @@ assert getattr(B_mro(), 'a', None) == None def test_abstract_mro(self): - class A1: - __metaclass__ = _classobj + class A1: # old-style class + pass class B1(A1): pass class C1(A1): @@ -595,7 +576,7 @@ def test_only_classic_bases_fails(self): class C: - __metaclass__ = _classobj + pass raises(TypeError, type, 'D', (C,), {}) def test_set___class__(self): Modified: pypy/dist/pypy/objspace/std/test/test_unicodeobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_unicodeobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_unicodeobject.py Sat Jul 12 14:55:43 2008 @@ -269,17 +269,10 @@ assert U(u'test').__class__ is U def test_call_unicode(self): - skip("does not work") - class X: + class X(object): def __unicode__(self): return u'x' - - try: - unicode(X(), 'ascii') - except TypeError, t: - assert 'need string or buffer' in str(t) - else: - raise Exception("DID NOT RAISE") + raises(TypeError, unicode, X(), 'ascii') def test_startswith(self): assert u'ab'.startswith(u'ab') is True Modified: pypy/dist/pypy/objspace/std/test/test_userobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_userobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_userobject.py Sat Jul 12 14:55:43 2008 @@ -8,21 +8,21 @@ cls.space = conftest.gettestobjspace(**cls.OPTIONS) def test_emptyclass(self): - class empty: pass + class empty(object): pass inst = empty() assert isinstance(inst, empty) - inst.attr=23 - assert inst.attr ==23 + inst.attr = 23 + assert inst.attr == 23 def test_method(self): - class A: + class A(object): def f(self, v): return v*42 a = A() assert a.f('?') == '??????????????????????????????????????????' def test_unboundmethod(self): - class A: + class A(object): def f(self, v): return v*17 a = A() @@ -46,7 +46,7 @@ assert len(result) ==3 def test_subsubclass(self): - class base: + class base(object): baseattr = 12 class derived(base): derivedattr = 34 @@ -56,7 +56,7 @@ assert inst.derivedattr ==34 def test_descr_get(self): - class C: + class C(object): class desc(object): def __get__(self, ob, cls=None): return 42 @@ -86,21 +86,21 @@ assert c.wibble == 22 def test_class_setattr(self): - class C: + class C(object): pass C.a = 1 assert hasattr(C, 'a') assert C.a == 1 def test_add(self): - class C: + class C(object): def __add__(self, other): return self, other c1 = C() assert c1+3 == (c1, 3) def test_call(self): - class C: + class C(object): def __call__(self, *args): return args c1 = C() @@ -118,7 +118,7 @@ assert c1.a == '->a' def test_getattr(self): - class C: + class C(object): def __getattr__(self, name): return '->' + name c1 = C() @@ -207,12 +207,6 @@ s = repr(Foo()) assert s.startswith('> sys.stderr, msg @@ -221,7 +216,6 @@ i = 0 run_module = False run_stdin = False - oldstyle_classes = False unbuffered = False while i < len(argv): arg = argv[i] @@ -260,8 +254,6 @@ return 2 run_module = True break - elif arg in ('-k', '--oldstyle'): - oldstyle_classes = True elif arg == '--': i += 1 break # terminates option list @@ -287,10 +279,6 @@ mainmodule = type(sys)('__main__') sys.modules['__main__'] = mainmodule - if oldstyle_classes: - import __builtin__ - __builtin__.__metaclass__ = __builtin__._classobj - if import_site: try: import site From arigo at codespeak.net Sat Jul 12 14:55:55 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 14:55:55 +0200 (CEST) Subject: [pypy-svn] r56490 - pypy/branch/always-oldstyle Message-ID: <20080712125555.AC42E2A8097@codespeak.net> Author: arigo Date: Sat Jul 12 14:55:55 2008 New Revision: 56490 Removed: pypy/branch/always-oldstyle/ Log: Kill merged branch. From arigo at codespeak.net Sat Jul 12 15:11:30 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 15:11:30 +0200 (CEST) Subject: [pypy-svn] r56491 - pypy/dist/pypy/module/__builtin__ Message-ID: <20080712131130.A4AF1169F8A@codespeak.net> Author: arigo Date: Sat Jul 12 15:11:29 2008 New Revision: 56491 Modified: pypy/dist/pypy/module/__builtin__/importing.py Log: Quick hack so that the function really does what its docstring suggests. Modified: pypy/dist/pypy/module/__builtin__/importing.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/importing.py (original) +++ pypy/dist/pypy/module/__builtin__/importing.py Sat Jul 12 15:11:29 2008 @@ -49,12 +49,12 @@ """ This is the way pypy does it. A pyc is only used if the py file exists AND the pyc file contains the timestamp of the py. """ pyfile_exist, pycfile_exists, pycfile_ts_valid = info_modtype(space, filepart) - if pycfile_ts_valid: + if not pyfile_exist: + return NOFILE + elif pycfile_ts_valid: return PYCFILE - elif pyfile_exist: - return PYFILE else: - return NOFILE + return PYFILE def find_modtype_cpython(space, filepart): """ This is the way cpython does it (where the py file doesnt exist but there From jlg at codespeak.net Sat Jul 12 15:20:03 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Sat, 12 Jul 2008 15:20:03 +0200 (CEST) Subject: [pypy-svn] r56493 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080712132003.C57C416A006@codespeak.net> Author: jlg Date: Sat Jul 12 15:20:03 2008 New Revision: 56493 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: interp_itertools - groupby consumes previous group when when next() called Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Sat Jul 12 15:20:03 2008 @@ -731,7 +731,8 @@ self.lookahead = False self.exhausted = False self.started = False - self.group_edge = True + # new_group - new group not started yet, next should not skip any items + self.new_group = True self.w_lookahead = self.space.w_None self.w_key = self.space.w_None @@ -742,6 +743,17 @@ if self.exhausted: raise OperationError(self.space.w_StopIteration, self.space.w_None) + if not self.new_group: + # Consume unwanted input until we reach the next group + try: + while True: + self.group_next(self.index) + + except StopIteration: + pass + if self.exhausted: + raise OperationError(self.space.w_StopIteration, self.space.w_None) + if not self.started: self.started = True try: @@ -758,20 +770,11 @@ self.w_key = self.space.call_function(self.w_fun, w_obj) self.lookahead = True - if not self.group_edge: - # Consume unwanted input until we reach the next group - try: - while True: - self.group_next(self.index) - except StopIteration: - pass - if self.exhausted: - raise OperationError(self.space.w_StopIteration, self.space.w_None) + self.new_group = False w_iterator = self.space.wrap(W_GroupByIterator(self.space, self.index, self)) return self.space.newtuple([self.w_key, w_iterator]) def group_next(self, group_index): - self.group_edge = False if group_index < self.index: raise StopIteration else: @@ -799,7 +802,7 @@ self.w_lookahead = w_obj self.w_key = w_new_key self.lookahead = True - self.group_edge = True + self.new_group = True #new group raise StopIteration def W_GroupBy___new__(space, w_subtype, w_iterable, w_fun=None): Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Sat Jul 12 15:20:03 2008 @@ -434,13 +434,31 @@ raises(StopIteration, g.next) raises(StopIteration, it.next) + # consumes after group started it = itertools.groupby([0, 0, 0, 0, 1]) k1, g1 = it.next() + assert g1.next() == 0 k2, g2 = it.next() raises(StopIteration, g1.next) assert g2.next() == 1 raises(StopIteration, g2.next) + # skips with not started group + it = itertools.groupby([0, 0, 1]) + k1, g1 = it.next() + k2, g2 = it.next() + raises(StopIteration, g1.next) + assert g2.next() == 1 + raises(StopIteration, g2.next) + + it = itertools.groupby([0, 1, 2]) + k1, g1 = it.next() + k2, g2 = it.next() + k2, g3 = it.next() + raises(StopIteration, g1.next) + raises(StopIteration, g2.next) + assert g3.next() == 2 + def half_floor(x): return x // 2 it = itertools.groupby([0, 1, 2, 3, 4, 5], half_floor) From hpk at codespeak.net Sat Jul 12 15:58:28 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sat, 12 Jul 2008 15:58:28 +0200 (CEST) Subject: [pypy-svn] r56494 - pypy/extradoc/planning/1.1 Message-ID: <20080712135828.D2815168545@codespeak.net> Author: hpk Date: Sat Jul 12 15:58:26 2008 New Revision: 56494 Modified: pypy/extradoc/planning/1.1/plan.txt Log: avoid alarming keyword Modified: pypy/extradoc/planning/1.1/plan.txt ============================================================================== --- pypy/extradoc/planning/1.1/plan.txt (original) +++ pypy/extradoc/planning/1.1/plan.txt Sat Jul 12 15:58:26 2008 @@ -17,7 +17,7 @@ - review ctypes skipped tests - threading works nicely - a build process that integrates well with distutils, - easy_install should fail gracefully for c-extensions + should fail gracefully for c-extensions - setup automated windows testing, integrate testing "pypy-c" on all platforms, **have one central pypy-c testing page** - check missing standard library modules From jlg at codespeak.net Sat Jul 12 16:25:11 2008 From: jlg at codespeak.net (jlg at codespeak.net) Date: Sat, 12 Jul 2008 16:25:11 +0200 (CEST) Subject: [pypy-svn] r56500 - pypy/dist/pypy/module/itertools/test Message-ID: <20080712142511.7DD84168549@codespeak.net> Author: jlg Date: Sat Jul 12 16:25:09 2008 New Revision: 56500 Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py Log: inter_itertools - groupby tests for wring arguments Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Sat Jul 12 16:25:09 2008 @@ -245,7 +245,7 @@ raises(StopIteration, it.next) def test_chain_wrongargs(self): - import itertools, re + import itertools raises(TypeError, itertools.chain, None) raises(TypeError, itertools.chain, [], None) @@ -256,7 +256,7 @@ try: itertools.chain(*args) except TypeError, e: - assert re.search(r'\b%d\b' % (x + 1), str(e)) + assert str(e).find("#" + str(x + 1) + " ") >= 0 else: fail("TypeError expected") @@ -344,7 +344,7 @@ try: itertools.izip(*args) except TypeError, e: - assert re.search(r'\b%d\b' % (x + 1), str(e)) + assert str(e).find("#" + str(x + 1) + " ") >= 0 else: fail("TypeError expected") @@ -497,6 +497,13 @@ raises(StopIteration, g.next) raises(StopIteration, it.next) + def test_groupby_wrongargs(self): + import itertools + + raises(TypeError, itertools.groupby, 0) + it = itertools.groupby([0], 1) + raises(TypeError, it.next) + def test_iterables(self): import itertools From antocuni at codespeak.net Sat Jul 12 16:26:50 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Sat, 12 Jul 2008 16:26:50 +0200 (CEST) Subject: [pypy-svn] r56501 - pypy/branch/ooexternals Message-ID: <20080712142650.450972A8090@codespeak.net> Author: antocuni Date: Sat Jul 12 16:26:48 2008 New Revision: 56501 Removed: pypy/branch/ooexternals/ Log: remove dead branch From arigo at codespeak.net Sat Jul 12 16:40:44 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 16:40:44 +0200 (CEST) Subject: [pypy-svn] r56502 - pypy/dist/pypy/doc Message-ID: <20080712144044.4D00969805F@codespeak.net> Author: arigo Date: Sat Jul 12 16:40:42 2008 New Revision: 56502 Modified: pypy/dist/pypy/doc/getting-started.txt Log: As usual we didn't get around to remove --faassen yet. At least we can stop documenting it and mention --allopts instead. Modified: pypy/dist/pypy/doc/getting-started.txt ============================================================================== --- pypy/dist/pypy/doc/getting-started.txt (original) +++ pypy/dist/pypy/doc/getting-started.txt Sat Jul 12 16:40:42 2008 @@ -551,7 +551,7 @@ nowadays is:: cd pypy/translator/goal - python translate.py --gc=hybrid --thread targetpypystandalone.py --allworkingmodules --faassen + python translate.py --gc=hybrid --thread targetpypystandalone.py --allworkingmodules --allopts Dependencies: this will compile all supported built-in modules, some of which have external dependencies. On a Debian Linux, for example, you @@ -566,7 +566,7 @@ exact generational garbage collector implementation, whose performance is rather good nowadays. The ``--thread`` option enables the thread module, which is still slightly experimental. -The ``--faassen`` option enables all the +The ``--allopts`` option enables all the worthwhile performance optimizations, but slows down the translation itself. On Linux 32-bit Intel machines, if you don't need threads, you can get some extra speed (and extra translation time) by removing From arigo at codespeak.net Sat Jul 12 16:45:22 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 16:45:22 +0200 (CEST) Subject: [pypy-svn] r56503 - pypy/dist/pypy/objspace/fake Message-ID: <20080712144522.0D773169F56@codespeak.net> Author: arigo Date: Sat Jul 12 16:45:21 2008 New Revision: 56503 Modified: pypy/dist/pypy/objspace/fake/objspace.py Log: Disable shortcuts for the fake objspace. Modified: pypy/dist/pypy/objspace/fake/objspace.py ============================================================================== --- pypy/dist/pypy/objspace/fake/objspace.py (original) +++ pypy/dist/pypy/objspace/fake/objspace.py Sat Jul 12 16:45:21 2008 @@ -49,6 +49,7 @@ def initialize(self): self.config.objspace.geninterp = False + self.config.objspace.disable_call_speedhacks = True self.wrap_cache = {} self.make_builtins() From arigo at codespeak.net Sat Jul 12 16:55:23 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 16:55:23 +0200 (CEST) Subject: [pypy-svn] r56504 - in pypy/dist/pypy: config module/__builtin__ module/__builtin__/test module/_file rlib Message-ID: <20080712145523.B3DA5169F59@codespeak.net> Author: arigo Date: Sat Jul 12 16:55:22 2008 New Revision: 56504 Modified: pypy/dist/pypy/config/pypyoption.py pypy/dist/pypy/module/__builtin__/importing.py pypy/dist/pypy/module/__builtin__/test/test_import.py pypy/dist/pypy/module/_file/interp_stream.py pypy/dist/pypy/rlib/streamio.py Log: Add an option lonepycfiles, false by default, which allows .pyc files to be imported even if the corresponding .py file does not exist. Sanitize some small parts of the import logic and try to make it robust against I/O errors. Modified: pypy/dist/pypy/config/pypyoption.py ============================================================================== --- pypy/dist/pypy/config/pypyoption.py (original) +++ pypy/dist/pypy/config/pypyoption.py Sat Jul 12 16:55:22 2008 @@ -137,7 +137,11 @@ BoolOption("usepycfiles", "Write and read pyc files when importing", default=True), - + + BoolOption("lonepycfiles", "Import pyc files with no matching py file", + default=False, + requires=[("objspace.usepycfiles", True)]), + BoolOption("honor__builtins__", "Honor the __builtins__ key of a module dictionary", default=False), Modified: pypy/dist/pypy/module/__builtin__/importing.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/importing.py (original) +++ pypy/dist/pypy/module/__builtin__/importing.py Sat Jul 12 16:55:22 2008 @@ -10,6 +10,7 @@ from pypy.interpreter.baseobjspace import W_Root, ObjSpace from pypy.interpreter.eval import Code from pypy.rlib import streamio +from pypy.rlib.streamio import StreamErrors from pypy.rlib.rarithmetic import intmask from pypy.rlib.objectmodel import we_are_translated @@ -17,55 +18,34 @@ PYFILE = 1 PYCFILE = 2 -def info_modtype(space, filepart): - """ - calculate whether the .py file exists, the .pyc file exists - and whether the .pyc file has the correct mtime entry. - The latter is only true if the .py file exists. - The .pyc file is only considered existing if it has a valid - magic number. +def find_modtype(space, filepart): + """Check which kind of module to import for the given filepart, + which is a path without extension. Returns PYFILE, PYCFILE or + NOFILE. """ + # check the .py file pyfile = filepart + ".py" - pyfile_exist = False if os.path.exists(pyfile): pyfile_ts = os.stat(pyfile)[stat.ST_MTIME] - pyfile_exist = True + pyfile_exists = True else: + # The .py file does not exist. By default on PyPy, lonepycfiles + # is False: if a .py file does not exist, we don't even try to + # look for a lone .pyc file. + if not space.config.objspace.lonepycfiles: + return NOFILE pyfile_ts = 0 - pyfile_exist = False - - pycfile = filepart + ".pyc" - if space.config.objspace.usepycfiles and os.path.exists(pycfile): - pyc_state = check_compiled_module(space, pyfile, pyfile_ts, pycfile) - pycfile_exists = pyc_state >= 0 - pycfile_ts_valid = pyc_state > 0 or (pyc_state == 0 and not pyfile_exist) - else: - pycfile_exists = False - pycfile_ts_valid = False - - return pyfile_exist, pycfile_exists, pycfile_ts_valid + pyfile_exists = False -def find_modtype(space, filepart): - """ This is the way pypy does it. A pyc is only used if the py file exists AND - the pyc file contains the timestamp of the py. """ - pyfile_exist, pycfile_exists, pycfile_ts_valid = info_modtype(space, filepart) - if not pyfile_exist: - return NOFILE - elif pycfile_ts_valid: - return PYCFILE - else: - return PYFILE - -def find_modtype_cpython(space, filepart): - """ This is the way cpython does it (where the py file doesnt exist but there - is a valid pyc file. """ - pyfile_exist, pycfile_exists, pycfile_ts_valid = info_modtype(space, filepart) - if pycfile_ts_valid: - return PYCFILE - elif pyfile_exist: + # check the .pyc file + if space.config.objspace.usepycfiles: + pycfile = filepart + ".pyc" + if check_compiled_module(space, pycfile, pyfile_ts): + return PYCFILE # existing and up-to-date .pyc file + + # no .pyc file, use the .py file if it exists + if pyfile_exists: return PYFILE - elif pycfile_exists: - return PYCFILE else: return NOFILE @@ -88,38 +68,40 @@ w = space.wrap w_mod = w(Module(space, w_modulename)) - e = None - if modtype == PYFILE: - filename = filepart + ".py" - stream = streamio.open_file_as_stream(filename, "rU") - else: - assert modtype == PYCFILE - filename = filepart + ".pyc" - stream = streamio.open_file_as_stream(filename, "rb") - - _prepare_module(space, w_mod, filename, pkgdir) try: + if modtype == PYFILE: + filename = filepart + ".py" + stream = streamio.open_file_as_stream(filename, "rU") + else: + assert modtype == PYCFILE + filename = filepart + ".pyc" + stream = streamio.open_file_as_stream(filename, "rb") + try: - if modtype == PYFILE: - load_source_module(space, w_modulename, w_mod, filename, stream.readall()) - else: - magic = _r_long(stream) - timestamp = _r_long(stream) - load_compiled_module(space, w_modulename, w_mod, filename, - magic, timestamp, stream.readall()) + _prepare_module(space, w_mod, filename, pkgdir) + try: + if modtype == PYFILE: + load_source_module(space, w_modulename, w_mod, filename, + stream.readall()) + else: + magic = _r_long(stream) + timestamp = _r_long(stream) + load_compiled_module(space, w_modulename, w_mod, filename, + magic, timestamp, stream.readall()) + + except OperationError, e: + w_mods = space.sys.get('modules') + space.call_method(w_mods,'pop', w_modulename, space.w_None) + raise finally: stream.close() - - except OperationError, e: - w_mods = space.sys.get('modules') - space.call_method(w_mods,'pop', w_modulename, space.w_None) - raise - + + except StreamErrors: + return None + w_mod = check_sys_modules(space, w_modulename) if w_mod is not None and w_parent is not None: space.setattr(w_parent, w_name, w_mod) - if e: - raise e return w_mod def try_getattr(space, w_obj, w_name): @@ -433,22 +415,25 @@ return w_mod def _get_long(s): - if len(s) < 4: - return -1 # good enough for our purposes a = ord(s[0]) b = ord(s[1]) c = ord(s[2]) d = ord(s[3]) - x = a | (b<<8) | (c<<16) | (d<<24) - if _r_correction and d & 0x80 and x > 0: - x -= _r_correction - return int(x) - -# helper, to avoid exposing internals of marshal and the -# difficulties of using it though applevel. -_r_correction = intmask(1L<<32) # == 0 on 32-bit machines + if d >= 0x80: + d -= 0x100 + return a | (b<<8) | (c<<16) | (d<<24) + +def _read_n(stream, n): + buf = '' + while len(buf) < n: + data = stream.read(n - len(buf)) + if not data: + raise streamio.StreamError("end of file") + buf += data + return buf + def _r_long(stream): - s = stream.read(4) # XXX XXX could return smaller string + s = _read_n(stream, 4) return _get_long(s) def _w_long(stream, x): @@ -461,27 +446,25 @@ d = x & 0xff stream.write(chr(a) + chr(b) + chr(c) + chr(d)) -def check_compiled_module(space, pathname, mtime, cpathname): +def check_compiled_module(space, pycfilename, expected_mtime=0): """ - Given a pathname for a Python source file, its time of last - modification, and a pathname for a compiled file, check whether the - compiled file represents the same version of the source. If so, - return a FILE pointer for the compiled file, positioned just after - the header; if not, return NULL. - Doesn't set an exception. + Check if a pyc file's magic number and (optionally) mtime match. """ - w_marshal = space.getbuiltinmodule('marshal') - stream = streamio.open_file_as_stream(cpathname, "rb") - magic = _r_long(stream) try: - if magic != get_pyc_magic(space): - return -1 - pyc_mtime = _r_long(stream) - if pyc_mtime != mtime: - return 0 - finally: - stream.close() - return 1 + stream = streamio.open_file_as_stream(pycfilename, "rb") + try: + magic = _r_long(stream) + if magic != get_pyc_magic(space): + return False + if expected_mtime != 0: + pyc_mtime = _r_long(stream) + if pyc_mtime != expected_mtime: + return False + finally: + stream.close() + except StreamErrors: + return False + return True def read_compiled_module(space, cpathname, strbuf): """ Read a code object from a file and check it for validity """ @@ -540,7 +523,7 @@ # try: stream = streamio.open_file_as_stream(cpathname, "wb") - except OSError: + except StreamErrors: return # cannot create file try: try: @@ -556,7 +539,7 @@ _w_long(stream, mtime) finally: stream.close() - except OSError: + except StreamErrors: try: os.unlink(cpathname) except OSError: Modified: pypy/dist/pypy/module/__builtin__/test/test_import.py ============================================================================== --- pypy/dist/pypy/module/__builtin__/test/test_import.py (original) +++ pypy/dist/pypy/module/__builtin__/test/test_import.py Sat Jul 12 16:55:22 2008 @@ -78,6 +78,9 @@ stream.readall()) finally: stream.close() + if space.config.objspace.usepycfiles: + # also create a lone .pyc file + p.join('lone.pyc').write(p.join('x.pyc').read()) return str(root) @@ -297,44 +300,38 @@ def test_check_compiled_module(self): space = self.space - pathname = "whatever" mtime = 12345 cpathname = _testfile(importing.get_pyc_magic(space), mtime) ret = importing.check_compiled_module(space, - pathname, - mtime, - cpathname) - assert ret == 1 + cpathname, + mtime) + assert ret is True # check for wrong mtime ret = importing.check_compiled_module(space, - pathname, - mtime+1, - cpathname) - assert ret == 0 + cpathname, + mtime+1) + assert ret is False os.remove(cpathname) # check for wrong version cpathname = _testfile(importing.get_pyc_magic(space)+1, mtime) ret = importing.check_compiled_module(space, - pathname, - mtime, - cpathname) - assert ret == -1 + cpathname, + mtime) + assert ret is False # check for empty .pyc file f = open(cpathname, 'wb') f.close() ret = importing.check_compiled_module(space, - pathname, - mtime, - cpathname) - assert ret == -1 + cpathname, + mtime) + assert ret is False os.remove(cpathname) def test_read_compiled_module(self): space = self.space - pathname = "whatever" mtime = 12345 co = compile('x = 42', '?', 'exec') cpathname = _testfile(importing.get_pyc_magic(space), mtime, co) @@ -355,7 +352,6 @@ def test_load_compiled_module(self): space = self.space - pathname = "whatever" mtime = 12345 co = compile('x = 42', '?', 'exec') cpathname = _testfile(importing.get_pyc_magic(space), mtime, co) @@ -522,12 +518,10 @@ mtime) # check - pathname = str(udir.join('cpathname.py')) ret = importing.check_compiled_module(space, - pathname, - mtime, - cpathname) - assert ret == 1 + cpathname, + mtime) + assert ret is True # read compiled module stream = streamio.open_file_as_stream(cpathname, "rb") @@ -642,3 +636,41 @@ sys.path.pop(0) sys.path.pop(0) sys.path_hooks.pop() + +class AppTestNoPycFile(object): + usepycfiles = False + lonepycfiles = False + + def setup_class(cls): + cls.space = gettestobjspace(**{ + "objspace.usepycfiles": cls.usepycfiles, + "objspace.lonepycfiles": cls.lonepycfiles, + }) + cls.w_usepycfiles = cls.space.wrap(cls.usepycfiles) + cls.w_lonepycfiles = cls.space.wrap(cls.lonepycfiles) + cls.saved_modules = _setup(cls.space) + + def teardown_class(cls): + _teardown(cls.space, cls.saved_modules) + + def test_import_possibly_from_pyc(self): + from compiled import x + if self.usepycfiles: + assert x.__file__.endswith('x.pyc') + else: + assert x.__file__.endswith('x.py') + try: + from compiled import lone + except ImportError: + assert not self.lonepycfiles, "should have found 'lone.pyc'" + else: + assert self.lonepycfiles, "should not have found 'lone.pyc'" + assert lone.__file__.endswith('lone.pyc') + +class AppTestNoLonePycFile(AppTestNoPycFile): + usepycfiles = True + lonepycfiles = False + +class AppTestLonePycFile(AppTestNoPycFile): + usepycfiles = True + lonepycfiles = True Modified: pypy/dist/pypy/module/_file/interp_stream.py ============================================================================== --- pypy/dist/pypy/module/_file/interp_stream.py (original) +++ pypy/dist/pypy/module/_file/interp_stream.py Sat Jul 12 16:55:22 2008 @@ -1,5 +1,6 @@ import py from pypy.rlib import streamio +from pypy.rlib.streamio import StreamErrors from errno import EINTR from pypy.interpreter.error import OperationError @@ -10,8 +11,6 @@ import os -StreamErrors = (OSError, streamio.StreamError) - def wrap_streamerror(space, e): if isinstance(e, streamio.StreamError): return OperationError(space.w_ValueError, Modified: pypy/dist/pypy/rlib/streamio.py ============================================================================== --- pypy/dist/pypy/rlib/streamio.py (original) +++ pypy/dist/pypy/rlib/streamio.py Sat Jul 12 16:55:22 2008 @@ -154,6 +154,8 @@ def __init__(self, message): self.message = message +StreamErrors = (OSError, StreamError) # errors that can generally be raised + class Stream(object): From arigo at codespeak.net Sat Jul 12 18:39:41 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 18:39:41 +0200 (CEST) Subject: [pypy-svn] r56505 - in pypy/dist/pypy/objspace/std: . test Message-ID: <20080712163941.E3591169F7B@codespeak.net> Author: arigo Date: Sat Jul 12 18:39:40 2008 New Revision: 56505 Modified: pypy/dist/pypy/objspace/std/floatobject.py pypy/dist/pypy/objspace/std/test/test_longobject.py Log: issue384 in-progress (pedronis, arigo) For now, remove the int-float comparison shortcuts. In addition to the issue of precision, going via floats for int-long comparisons looks very strange. Most importantly this fixes the bug shown in test_longobject.py. Modified: pypy/dist/pypy/objspace/std/floatobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/floatobject.py (original) +++ pypy/dist/pypy/objspace/std/floatobject.py Sat Jul 12 18:39:40 2008 @@ -124,10 +124,11 @@ for op in ['lt', 'le', 'eq', 'ne', 'gt', 'ge']: func, name = declare_new_float_comparison(op) globals()[name] = func - func, name = declare_new_int_float_comparison(op) - globals()[name] = func - func, name = declare_new_float_int_comparison(op) - globals()[name] = func + # XXX shortcuts disabled: see r54171 and issue #384. + #func, name = declare_new_int_float_comparison(op) + #globals()[name] = func + #func, name = declare_new_float_int_comparison(op) + #globals()[name] = func # for overflowing comparisons between longs and floats # XXX we might have to worry (later) about eq__Float_Int, for the case Modified: pypy/dist/pypy/objspace/std/test/test_longobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_longobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_longobject.py Sat Jul 12 18:39:40 2008 @@ -60,7 +60,6 @@ assert a == 3L def test_compare(self): - skip("in-progress") BIG = 1L << 9999 assert 0 == 0L assert not (0 != 0L) From arigo at codespeak.net Sat Jul 12 18:45:54 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 12 Jul 2008 18:45:54 +0200 (CEST) Subject: [pypy-svn] r56506 - pypy/dist/pypy/objspace/std/test Message-ID: <20080712164554.98E44169F9E@codespeak.net> Author: arigo Date: Sat Jul 12 18:45:54 2008 New Revision: 56506 Modified: pypy/dist/pypy/objspace/std/test/test_floatobject.py Log: issue384 in-progress (pedronis, arigo) A test that fails if declare_new_int_float_comparison() is enabled. Modified: pypy/dist/pypy/objspace/std/test/test_floatobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/test/test_floatobject.py (original) +++ pypy/dist/pypy/objspace/std/test/test_floatobject.py Sat Jul 12 18:45:54 2008 @@ -184,3 +184,10 @@ assert 13 < 13.01 assert 13 <= 13.0 assert 13 <= 13.01 + + def test_multimethod_slice(self): + assert 5 .__add__(3.14) is NotImplemented + assert 3.25 .__add__(5) == 8.25 + if hasattr(int, '__eq__'): # for py.test -A: CPython is inconsistent + assert 5 .__eq__(3.14) is NotImplemented + assert 3.14 .__eq__(5) is False From hruske at codespeak.net Sun Jul 13 16:01:24 2008 From: hruske at codespeak.net (hruske at codespeak.net) Date: Sun, 13 Jul 2008 16:01:24 +0200 (CEST) Subject: [pypy-svn] r56511 - in pypy/dist/pypy/lib: . test2 Message-ID: <20080713140124.D2B28169EC0@codespeak.net> Author: hruske Date: Sun Jul 13 16:01:24 2008 New Revision: 56511 Added: pypy/dist/pypy/lib/_hashlib.py pypy/dist/pypy/lib/test2/test_hashlib.py Modified: pypy/dist/pypy/lib/hashlib.py Log: Adding my _hashlib implementation based on _hashopenssl.c, works on Linux, on OS X only limited functionality since the OS X bundled library only provides MD5 and SHA1 hashes. Also worth noting is that MD5 and SHA1 will always work. Also adding unmodified test_hashlib.py from Python 2.5 Added: pypy/dist/pypy/lib/_hashlib.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/_hashlib.py Sun Jul 13 16:01:24 2008 @@ -0,0 +1,160 @@ +from ctypes import * +import ctypes.util + +# Note: OpenSSL on OS X only provides md5 and sha1 +libpath = ctypes.util.find_library('ssl') +lib = CDLL(libpath) # Linux, OS X + + +# FIXME do we really need this anywhere here? +class ENV_MD(Structure): + # XXX Should there be more to this object?. + _fields_ = [ + ('type', c_int), + ('pkey_type', c_int), + ('md_size', c_int), + ] + +class _dummy_env_md(Structure): + # XXX used for OS X, a bit hairy + _fields_ = [ + ('digest', ENV_MD), + ('two', c_int), + ('three', c_int), + ('four', c_int), + ('five', c_int), + ] + +def _new_ENV_MD(): + return _dummy_env_md() + +# OpenSSL initialization +lib.OpenSSL_add_all_digests() + +def _get_digest(ctx): + return ctx.digest + +# taken from evp.h, max size is 512 bit, 64 chars +lib.EVP_MAX_MD_SIZE = 64 + +class hash(object): + """ + A hash represents the object used to calculate a checksum of a + string of information. + + Methods: + + update() -- updates the current digest with an additional string + digest() -- return the current digest value + hexdigest() -- return the current digest as a string of hexadecimal digits + copy() -- return a copy of the current hash object + + Attributes: + + name -- the hash algorithm being used by this object + digest_size -- number of bytes in this hashes output + """ + def __init__(self, obj, name): + self.name = name # part of API + #print 'obj is ', obj + if isinstance(obj, _dummy_env_md): + self._obj = obj.digest + else: + self._obj = obj # private + + def __repr__(self): + # format is not the same as in C module + return "<%s HASH object>" % (self.name) + + def copy(self): + "Return a copy of the hash object." + ctxnew = _new_ENV_MD() + lib.EVP_MD_CTX_copy(byref(ctxnew), byref(self._obj)) + return hash(ctxnew, self.name) + + def hexdigest(self): + "Return the digest value as a string of hexadecimal digits." + dig = self.digest() + a = [] + for x in dig: + a.append('%.2x' % ord(x)) + #print '\n--- %r \n' % ''.join(a) + return ''.join(a) + + def digest(self): + "Return the digest value as a string of binary data." + tmpctx = self.copy() + digest_size = tmpctx.digest_size + dig = create_string_buffer(lib.EVP_MAX_MD_SIZE) + lib.EVP_DigestFinal(byref(tmpctx._obj), dig, None) + lib.EVP_MD_CTX_cleanup(byref(tmpctx._obj)) + return dig.raw[:digest_size] + + def digest_size(self): + # XXX This isn't the nicest way, but the EVP_MD_size OpenSSL function + # XXX is defined as a C macro on OS X and would be significantly + # XXX harder to implement in another way. + # Values are digest sizes in bytes + return { + 'md5': 16, + 'sha1': 20, + 'sha224': 28, + 'sha256': 32, + 'sha384': 48, + 'sha512': 64, + }.get(self.name, 0) + digest_size = property(digest_size, None, None) # PEP 247 + digestsize = digest_size # deprecated, was once defined by sha module + + def block_size(self): + return lib.EVP_MD_CTX_block_size(byref(self._obj)) + block_size = property(block_size, None, None) + + def update(self, string): + "Update this hash object's state with the provided string." + lib.EVP_DigestUpdate(byref(self._obj), c_char_p(string), c_uint(len(string))) + +def new(name, string=''): + """ + Return a new hash object using the named algorithm. + An optional string argument may be provided and will be + automatically hashed. + + The MD5 and SHA1 algorithms are always supported. + """ + digest = lib.EVP_get_digestbyname(c_char_p(name)) + + if not isinstance(name, str): + raise TypeError("name must be a string") + if not digest: + raise ValueError("unknown hash function") + + ctx = _new_ENV_MD() + lib.EVP_DigestInit(pointer(ctx), digest) + + h = hash(_get_digest(ctx), name) + if string: + if not isinstance(string, str): + raise ValueError("hash content is not string") + h.update(string) + return hash(ctx, name) + +# shortcut functions +def openssl_md5(string=''): + return new('md5', string) + +def openssl_sha1(string=''): + return new('sha1', string) + +def openssl_sha224(string=''): + return new('sha224', string) + +def openssl_sha256(string=''): + return new('sha256', string) + +def openssl_sha384(string=''): + return new('sha384', string) + +def openssl_sha512(string=''): + return new('sha512', string) + Modified: pypy/dist/pypy/lib/hashlib.py ============================================================================== --- pypy/dist/pypy/lib/hashlib.py (original) +++ pypy/dist/pypy/lib/hashlib.py Sun Jul 13 16:01:24 2008 @@ -114,6 +114,7 @@ del funcName except ImportError: + raise # XXX Don't try to load nonexistent C modules on PyPy # We don't have the _hashlib OpenSSL module? # use the built in legacy interfaces via a wrapper function new = __py_new @@ -121,7 +122,7 @@ # lookup the C function to use directly for the named constructors md5 = __get_builtin_constructor('md5') sha1 = __get_builtin_constructor('sha1') - #sha224 = __get_builtin_constructor('sha224') - #sha256 = __get_builtin_constructor('sha256') - #sha384 = __get_builtin_constructor('sha384') - #sha512 = __get_builtin_constructor('sha512') + sha224 = __get_builtin_constructor('sha224') + sha256 = __get_builtin_constructor('sha256') + sha384 = __get_builtin_constructor('sha384') + sha512 = __get_builtin_constructor('sha512') Added: pypy/dist/pypy/lib/test2/test_hashlib.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lib/test2/test_hashlib.py Sun Jul 13 16:01:24 2008 @@ -0,0 +1,191 @@ +# Test hashlib module +# +# $Id: test_hashlib.py 39316 2005-08-21 18:45:59Z greg $ +# +# Copyright (C) 2005 Gregory P. Smith (greg at electricrain.com) +# Licensed to PSF under a Contributor Agreement. +# + +import hashlib +import unittest +from test import test_support + + +def hexstr(s): + import string + h = string.hexdigits + r = '' + for c in s: + i = ord(c) + r = r + h[(i >> 4) & 0xF] + h[i & 0xF] + return r + + +class HashLibTestCase(unittest.TestCase): + supported_hash_names = ( 'md5', 'MD5', 'sha1', 'SHA1', + 'sha224', 'SHA224', 'sha256', 'SHA256', + 'sha384', 'SHA384', 'sha512', 'SHA512' ) + + def test_unknown_hash(self): + try: + hashlib.new('spam spam spam spam spam') + except ValueError: + pass + else: + self.assert_(0 == "hashlib didn't reject bogus hash name") + + def test_hexdigest(self): + for name in self.supported_hash_names: + h = hashlib.new(name) + self.assert_(hexstr(h.digest()) == h.hexdigest()) + + + def test_large_update(self): + aas = 'a' * 128 + bees = 'b' * 127 + cees = 'c' * 126 + + for name in self.supported_hash_names: + m1 = hashlib.new(name) + m1.update(aas) + m1.update(bees) + m1.update(cees) + + m2 = hashlib.new(name) + m2.update(aas + bees + cees) + self.assertEqual(m1.digest(), m2.digest()) + + + def check(self, name, data, digest): + # test the direct constructors + computed = getattr(hashlib, name)(data).hexdigest() + self.assert_(computed == digest) + # test the general new() interface + computed = hashlib.new(name, data).hexdigest() + self.assert_(computed == digest) + + + def test_case_md5_0(self): + self.check('md5', '', 'd41d8cd98f00b204e9800998ecf8427e') + + def test_case_md5_1(self): + self.check('md5', 'abc', '900150983cd24fb0d6963f7d28e17f72') + + def test_case_md5_2(self): + self.check('md5', 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789', + 'd174ab98d277d9f5a5611c2c9f419d9f') + + + # use the three examples from Federal Information Processing Standards + # Publication 180-1, Secure Hash Standard, 1995 April 17 + # http://www.itl.nist.gov/div897/pubs/fip180-1.htm + + def test_case_sha1_0(self): + self.check('sha1', "", + "da39a3ee5e6b4b0d3255bfef95601890afd80709") + + def test_case_sha1_1(self): + self.check('sha1', "abc", + "a9993e364706816aba3e25717850c26c9cd0d89d") + + def test_case_sha1_2(self): + self.check('sha1', "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", + "84983e441c3bd26ebaae4aa1f95129e5e54670f1") + + def test_case_sha1_3(self): + self.check('sha1', "a" * 1000000, + "34aa973cd4c4daa4f61eeb2bdbad27316534016f") + + + # use the examples from Federal Information Processing Standards + # Publication 180-2, Secure Hash Standard, 2002 August 1 + # http://csrc.nist.gov/publications/fips/fips180-2/fips180-2.pdf + + def test_case_sha224_0(self): + self.check('sha224', "", + "d14a028c2a3a2bc9476102bb288234c415a2b01f828ea62ac5b3e42f") + + def test_case_sha224_1(self): + self.check('sha224', "abc", + "23097d223405d8228642a477bda255b32aadbce4bda0b3f7e36c9da7") + + def test_case_sha224_2(self): + self.check('sha224', + "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", + "75388b16512776cc5dba5da1fd890150b0c6455cb4f58b1952522525") + + def test_case_sha224_3(self): + self.check('sha224', "a" * 1000000, + "20794655980c91d8bbb4c1ea97618a4bf03f42581948b2ee4ee7ad67") + + + def test_case_sha256_0(self): + self.check('sha256', "", + "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855") + + def test_case_sha256_1(self): + self.check('sha256', "abc", + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad") + + def test_case_sha256_2(self): + self.check('sha256', + "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", + "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1") + + def test_case_sha256_3(self): + self.check('sha256', "a" * 1000000, + "cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0") + + + def test_case_sha384_0(self): + self.check('sha384', "", + "38b060a751ac96384cd9327eb1b1e36a21fdb71114be07434c0cc7bf63f6e1da"+ + "274edebfe76f65fbd51ad2f14898b95b") + + def test_case_sha384_1(self): + self.check('sha384', "abc", + "cb00753f45a35e8bb5a03d699ac65007272c32ab0eded1631a8b605a43ff5bed"+ + "8086072ba1e7cc2358baeca134c825a7") + + def test_case_sha384_2(self): + self.check('sha384', + "abcdefghbcdefghicdefghijdefghijkefghijklfghijklmghijklmn"+ + "hijklmnoijklmnopjklmnopqklmnopqrlmnopqrsmnopqrstnopqrstu", + "09330c33f71147e83d192fc782cd1b4753111b173b3b05d22fa08086e3b0f712"+ + "fcc7c71a557e2db966c3e9fa91746039") + + def test_case_sha384_3(self): + self.check('sha384', "a" * 1000000, + "9d0e1809716474cb086e834e310a4a1ced149e9c00f248527972cec5704c2a5b"+ + "07b8b3dc38ecc4ebae97ddd87f3d8985") + + + def test_case_sha512_0(self): + self.check('sha512', "", + "cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce"+ + "47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e") + + def test_case_sha512_1(self): + self.check('sha512', "abc", + "ddaf35a193617abacc417349ae20413112e6fa4e89a97ea20a9eeee64b55d39a"+ + "2192992a274fc1a836ba3c23a3feebbd454d4423643ce80e2a9ac94fa54ca49f") + + def test_case_sha512_2(self): + self.check('sha512', + "abcdefghbcdefghicdefghijdefghijkefghijklfghijklmghijklmn"+ + "hijklmnoijklmnopjklmnopqklmnopqrlmnopqrsmnopqrstnopqrstu", + "8e959b75dae313da8cf4f72814fc143f8f7779c6eb9f7fa17299aeadb6889018"+ + "501d289e4900f7e4331b99dec4b5433ac7d329eeb6dd26545e96e55b874be909") + + def test_case_sha512_3(self): + self.check('sha512', "a" * 1000000, + "e718483d0ce769644e2e42c7bc15b4638e1f98b13b2044285632a803afa973eb"+ + "de0ff244877ea60a4cb0432ce577c31beb009c5c2c49aa2e4eadb217ad8cc09b") + + +def test_main(): + test_support.run_unittest(HashLibTestCase) + + +if __name__ == "__main__": + test_main() From hpk at codespeak.net Sun Jul 13 19:23:00 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sun, 13 Jul 2008 19:23:00 +0200 (CEST) Subject: [pypy-svn] r56521 - in pypy/dist: lib-python lib-python/modified-2.4.1/test pypy/lib/test2 Message-ID: <20080713172300.7BBD22A0150@codespeak.net> Author: hpk Date: Sun Jul 13 19:23:00 2008 New Revision: 56521 Added: pypy/dist/lib-python/modified-2.4.1/test/test_hashlib.py - copied unchanged from r56513, pypy/dist/pypy/lib/test2/test_hashlib.py Removed: pypy/dist/pypy/lib/test2/test_hashlib.py Modified: pypy/dist/lib-python/conftest.py Log: hruskec: in pypy/lib/test2 we keep only extra tests and they need to be py.test style. So I moved the test to a more appropriate place. Modified: pypy/dist/lib-python/conftest.py ============================================================================== --- pypy/dist/lib-python/conftest.py (original) +++ pypy/dist/lib-python/conftest.py Sun Jul 13 19:23:00 2008 @@ -511,7 +511,7 @@ RegrTest('test_gzip.py', enabled=False, dumbtest=1), RegrTest('test_hash.py', enabled=True, core=True), RegrTest('test_hashlib.py', enabled=True, core=True), - # test_hashlib is stripped down, originates from 2.5 + # test_hashlib comes from 2.5 RegrTest('test_heapq.py', enabled=True, core=True), RegrTest('test_hexoct.py', enabled=True, core=True), From hpk at codespeak.net Sun Jul 13 20:07:39 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sun, 13 Jul 2008 20:07:39 +0200 (CEST) Subject: [pypy-svn] r56522 - pypy/dist/pypy/module/zipimport Message-ID: <20080713180739.74D0C2D800C@codespeak.net> Author: hpk Date: Sun Jul 13 20:07:37 2008 New Revision: 56522 Modified: pypy/dist/pypy/module/zipimport/interp_zipimport.py Log: remove unused import Modified: pypy/dist/pypy/module/zipimport/interp_zipimport.py ============================================================================== --- pypy/dist/pypy/module/zipimport/interp_zipimport.py (original) +++ pypy/dist/pypy/module/zipimport/interp_zipimport.py Sun Jul 13 20:07:37 2008 @@ -4,8 +4,6 @@ from pypy.interpreter.error import OperationError, wrap_oserror from pypy.interpreter.gateway import interp2app from pypy.interpreter.typedef import TypeDef, GetSetProperty -from pypy.interpreter.gateway import app2interp -from pypy.interpreter.eval import Code from pypy.interpreter.module import Module from pypy.module.__builtin__ import importing from pypy.rlib.unroll import unrolling_iterable From hpk at codespeak.net Sun Jul 13 20:40:08 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sun, 13 Jul 2008 20:40:08 +0200 (CEST) Subject: [pypy-svn] r56523 - pypy/dist/pypy/lib Message-ID: <20080713184008.D5BF6169F21@codespeak.net> Author: hpk Date: Sun Jul 13 20:40:05 2008 New Revision: 56523 Modified: pypy/dist/pypy/lib/hashlib.py Log: cleanup hashlib module, should still pass lib-python test_hashlib test Modified: pypy/dist/pypy/lib/hashlib.py ============================================================================== --- pypy/dist/pypy/lib/hashlib.py (original) +++ pypy/dist/pypy/lib/hashlib.py Sun Jul 13 20:40:05 2008 @@ -1,6 +1,3 @@ -# XXX ported version of the original hashlib.py from python -# XXX 2.5, only supports md5 and sha1 by using the existing -# XXX modules of PyPy # $Id: hashlib.py 52533 2006-10-29 18:01:12Z georg.brandl $ # # Copyright (C) 2005 Gregory P. Smith (greg at electricrain.com) @@ -62,17 +59,8 @@ elif name in ('MD5', 'md5'): import md5 return md5.new - # XXX code for other codecs deleted here raise ValueError, "unsupported hash type" - -def __py_new(name, string=''): - """new(name, string='') - Return a new hashing object using the named algorithm; - optionally initialized with a string. - """ - return __get_builtin_constructor(name)(string) - - def __hash_new(name, string=''): """new(name, string='') - Return a new hashing object using the named algorithm; optionally initialized with a string. @@ -87,42 +75,28 @@ return __get_builtin_constructor(name)(string) -try: +def _setfuncs(): import _hashlib # use the wrapper of the C implementation new = __hash_new - for opensslFuncName in filter(lambda n: n.startswith('openssl_'), dir(_hashlib)): - funcName = opensslFuncName[len('openssl_'):] + sslprefix = 'openssl_' + for opensslfuncname, func in vars(_hashlib).items(): + if not opensslfuncname.startswith(sslprefix): + continue + funcname = opensslfuncname[len(sslprefix):] try: # try them all, some may not work due to the OpenSSL # version not supporting that algorithm. - f = getattr(_hashlib, opensslFuncName) - f() + func() # Use the C function directly (very fast) - exec funcName + ' = f' + globals()[funcname] = func except ValueError: try: # Use the builtin implementation directly (fast) - exec funcName + ' = __get_builtin_constructor(funcName)' + globals()[funcname] = __get_builtin_constructor(funcname) except ValueError: # this one has no builtin implementation, don't define it pass - # clean up our locals - del f - del opensslFuncName - del funcName - -except ImportError: - raise # XXX Don't try to load nonexistent C modules on PyPy - # We don't have the _hashlib OpenSSL module? - # use the built in legacy interfaces via a wrapper function - new = __py_new - - # lookup the C function to use directly for the named constructors - md5 = __get_builtin_constructor('md5') - sha1 = __get_builtin_constructor('sha1') - sha224 = __get_builtin_constructor('sha224') - sha256 = __get_builtin_constructor('sha256') - sha384 = __get_builtin_constructor('sha384') - sha512 = __get_builtin_constructor('sha512') + +_setfuncs() From hpk at codespeak.net Sun Jul 13 20:43:24 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Sun, 13 Jul 2008 20:43:24 +0200 (CEST) Subject: [pypy-svn] r56524 - pypy/dist/pypy/lib Message-ID: <20080713184324.5E96D169F21@codespeak.net> Author: hpk Date: Sun Jul 13 20:43:23 2008 New Revision: 56524 Modified: pypy/dist/pypy/lib/hashlib.py Log: fix little issues. (aka should != does) Modified: pypy/dist/pypy/lib/hashlib.py ============================================================================== --- pypy/dist/pypy/lib/hashlib.py (original) +++ pypy/dist/pypy/lib/hashlib.py Sun Jul 13 20:43:23 2008 @@ -50,6 +50,7 @@ 'a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2' """ +import _hashlib def __get_builtin_constructor(name): @@ -74,11 +75,10 @@ # the OpenSSL library prior to 0.9.8 doesn't provide them. return __get_builtin_constructor(name)(string) +new = __hash_new def _setfuncs(): - import _hashlib # use the wrapper of the C implementation - new = __hash_new sslprefix = 'openssl_' for opensslfuncname, func in vars(_hashlib).items(): From hpk at codespeak.net Mon Jul 14 09:41:12 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Mon, 14 Jul 2008 09:41:12 +0200 (CEST) Subject: [pypy-svn] r56529 - pypy/extradoc/talk/ep2008 Message-ID: <20080714074112.6999E2A80A3@codespeak.net> Author: hpk Date: Mon Jul 14 09:41:09 2008 New Revision: 56529 Added: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt (contents, props changed) pypy/extradoc/talk/ep2008/ui (contents, props changed) Log: adding "behind the scene talk" Added: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt Mon Jul 14 09:41:09 2008 @@ -0,0 +1,191 @@ +.. include:: + +================================= +PyPy - Behind the Scenes +================================= + +:event: EuroPython 2008, Vilnius +:Authors: Holger Krekel & Bea During + +Who and how +--------------------- +- Holger Krekel: merlinux GmbH +- Bea During: Open End AB/Change Maker + +We aim to highlight development culture, +backgrounds, roadmap and funding topics. + +please interrupt and ask questions! + +The When of PyPy +--------------------- +- 2001: startup of PyPy OSS work on codespeak + (get your own JIT for your favourite language for free) +- 2004-2007: consolidation through EU-project + (results as well as growing community) +- 2008--> towards production ready through company funding + +Development culture +----------------------- +- Mix of agile and OSS practices +- Core practice is sprint-driven development +- We aim to have a low drop in level (sprints/IRC/mail) +- Mixed interest of being research and real world + usage driven + +It's all about scope +------------------------ + +Making things fit formula: + +Vision->Roadmap/scope->direction->results + +People->funding contribution and matching interests + +The Scope Game/EU-funding +--------------------------- + +Big design up front, Contractual framework organised via consortium + +plus: **Funding makes for more predictable results** + +minus: + +- Guessing game too far into the future + +- Reacting/incorporating community contribution + +- Adjusting direction based on community needs + +The Scope Game/Company funding (plus) +----------------------------------------- +Google - small increments of work (Ctypes, Django) +Contractually organised through PyPy core companies + +- Small increments->easier to incorporate community +- Small increments->flexibility regarding direction +- PyPy core companies set aside sprint funding +- Companies are need driven + +The Scope Game/Company funding (minus) +--------------------------------------- +- All work not suited for small increments +- Company interests not matching contributors interests +- Company interests not matching interests of wider community + +Examples: separate compilation, JIT work, +release work, refactoring etc + +The Scope Game/The right time is now +------------------------------------- +Find the overlap between: + +- community needs +- vision and direction of PyPy +- commercial needs + +Successful match of these means spot market, +longer collaborations means partnerships + +Technical status +-------------------------- + +* Compatibility: very good and getting better +* Speed: without JIT getting closer to CPython +* Features: threads and ctypes, more extmodules working +* PyPy well suited to target new platforms +* JIT: improving, but not ready + +Road map items and goals (1) +---------------------------------- + +- **pypy-c as cpython replacement** +- **robustness** +- **sandboxing** +- **interactivity / startup time** +- new build targets + + +Road map items and goals (2) +---------------------------- + +- Optimize memory resources +- free threading +- New OO backends / Tamarin +- **pypy.NET** +- **pypy.JVM** + +Road map items and goals (3) +---------------------------- + +- **JIT Compiler Generator** +- ECMA Javascript Interpreter +- **Pyrolog**: Prolog +- **SPY**: Smalltalk/squeak +- Gaming engines + +Next steps +---------------------------- +* Basis: Many individual contributors +* incremental proposals -> pypy-c +* longer term: + - PyJIT (Eurostars) + - "dynamic internet" (EU framework programme 7) + +Contribution and Funding +----------------------------------------- +- Start using pypy-c +- Write code, docs, send patches +- Organise or sponsor a sprint +- Goal oriented contracts +- write your PHD, thesis about PyPy +- educate using PyPy + + +Collaboration +----------------- +* Python Implementations: CPython & Jython developers / Sun +* Guido and Google open source proposals +* SPY (pypy sqeak/smalltalk): Bern University +* LLVM Chris Lattner, Anton Korbeynikov + +PyPy contributors +--------------------- + +Armin Rigo, Carl Friedrich Bolz, Maciej Fijalkowski +Antonio Cuni, Samuele Pedroni, Niko Matsakis, +Holger Krekel, Alexander Schremmer, Amaury Forgeot d Arc, +Christian Tismer, Jacob Hallen, Laura Creighton, Beatrice During, +Leonardo Santagada, Stephan Diehl, Lene Wagner, +Jean-Paul Calderone, Michael Schneider, Lene Wagner, +Michael Hudson, Richard Emslie, Eric van Riet Paap, +Aurelien Campeas, Nikolaus Haldimann, ... + +Involved Organisations +------------------------ + +**merlinux**: Holger Krekel, Maciej Fijalkowski, Antonio Cuni, Carl Friedrich Bolz, Lene Wagner + +**Open End**: Samuele Pedroni, Bea During, Jacob Hallen, Laura Creighton + +**Heinrich Heine University**: Carl Friedrich Bolz, Michael Leuschel, (Armin Rigo) + +**Tismerysoft**: Christian Tismer + +**University of Genova**: Antonio Cuni + +Contact, questions, discussion +------------------------------- + +Holger Krekel + +Beatrice During + +pypy-dev at codespeak.net +#pypy on irc.freenode.org + +http://codespeak.net/pypy + +http://merlinux.eu + +http://openend.se Added: pypy/extradoc/talk/ep2008/ui ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/ui Mon Jul 14 09:41:09 2008 @@ -0,0 +1 @@ +link ../ui \ No newline at end of file From hpk at codespeak.net Mon Jul 14 09:42:57 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Mon, 14 Jul 2008 09:42:57 +0200 (CEST) Subject: [pypy-svn] r56530 - pypy/extradoc/talk/ep2008 Message-ID: <20080714074257.40F3E2A807B@codespeak.net> Author: hpk Date: Mon Jul 14 09:42:54 2008 New Revision: 56530 Added: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.html Log: adding the html version Added: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.html ============================================================================== --- (empty file) +++ pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.html Mon Jul 14 09:42:54 2008 @@ -0,0 +1,536 @@ + + + + + + + +PyPy - Behind the Scenes + + + + + + + + + + + + + + + +

+
+
+ + +
+
+
+

PyPy - Behind the Scenes

+ +++ + + + + + +
event:EuroPython 2008, Vilnius
Authors:Holger Krekel & Bea During
+ + + + + + + + + + +
+
+

Who and how

+
    +
  • Holger Krekel: merlinux GmbH
    • +
  • Bea During: Open End AB/Change Maker
    • +
+

We aim to highlight development culture, +backgrounds, roadmap and funding topics.

+

please interrupt and ask questions!

+
+
+

The When of PyPy

+
    +
  • 2001: startup of PyPy OSS work on codespeak +(get your own JIT for your favourite language for free)
    • +
  • 2004-2007: consolidation through EU-project +(results as well as growing community)
    • +
  • 2008--> towards production ready through company funding
    • +
+
+
+

Development culture

+
    +
  • Mix of agile and OSS practices
    • +
  • Core practice is sprint-driven development
    • +
  • We aim to have a low drop in level (sprints/IRC/mail)
    • +
  • Mixed interest of being research and real world +usage driven
    • +
+
+
+

It's all about scope

+

Making things fit formula:

+

Vision->Roadmap/scope->direction->results

+

People->funding contribution and matching interests

+
+
+

The Scope Game/EU-funding

+

Big design up front, Contractual framework organised via consortium

+

plus: Funding makes for more predictable results

+

minus:

+
    +
  • Guessing game too far into the future
    • +
  • Reacting/incorporating community contribution
    • +
  • Adjusting direction based on community needs
    • +
+
+
+

The Scope Game/Company funding (plus)

+

Google - small increments of work (Ctypes, Django) +Contractually organised through PyPy core companies

+
    +
  • Small increments->easier to incorporate community
    • +
  • Small increments->flexibility regarding direction
    • +
  • PyPy core companies set aside sprint funding
    • +
  • Companies are need driven
    • +
+
+
+

The Scope Game/Company funding (minus)

+
    +
  • All work not suited for small increments
    • +
  • Company interests not matching contributors interests
    • +
  • Company interests not matching interests of wider community
    • +
+

Examples: separate compilation, JIT work, +release work, refactoring etc

+
+
+

The Scope Game/The right time is now

+

Find the overlap between:

+
    +
  • community needs
    • +
  • vision and direction of PyPy
    • +
  • commercial needs
    • +
+

Successful match of these means spot market, +longer collaborations means partnerships

+
+
+

Technical status

+
    +
  • Compatibility: very good and getting better
    • +
  • Speed: without JIT getting closer to CPython
    • +
  • Features: threads and ctypes, more extmodules working
    • +
  • PyPy well suited to target new platforms
    • +
  • JIT: improving, but not ready
    • +
+
+
+

Road map items and goals (1)

+
    +
  • pypy-c as cpython replacement
    • +
  • robustness
    • +
  • sandboxing
    • +
  • interactivity / startup time
    • +
  • new build targets
    • +
+
+
+

Road map items and goals (2)

+
    +
  • Optimize memory resources
    • +
  • free threading
    • +
  • New OO backends / Tamarin
    • +
  • pypy.NET
    • +
  • pypy.JVM
    • +
+
+
+

Road map items and goals (3)

+
    +
  • JIT Compiler Generator
    • +
  • ECMA Javascript Interpreter
    • +
  • Pyrolog: Prolog
    • +
  • SPY: Smalltalk/squeak
    • +
  • Gaming engines
    • +
+
+
+

Next steps

+
    +

  • Basis: Many individual contributors

    +
    • +

  • incremental proposals -> pypy-c

    +
    • +
  • +
    longer term:
    +
      +
    • PyJIT (Eurostars)
      • +
    • "dynamic internet" (EU framework programme 7)
      • +
    +
    +
    +
    • +
+
+
+

Contribution and Funding

+
    +
  • Start using pypy-c
    • +
  • Write code, docs, send patches
    • +
  • Organise or sponsor a sprint
    • +
  • Goal oriented contracts
    • +
  • write your PHD, thesis about PyPy
    • +
  • educate using PyPy
    • +
+
+
+

Collaboration

+
    +
  • Python Implementations: CPython & Jython developers / Sun
    • +
  • Guido and Google open source proposals
    • +
  • SPY (pypy sqeak/smalltalk): Bern University
    • +
  • LLVM Chris Lattner, Anton Korbeynikov
    • +
+
+
+

PyPy contributors

+

Armin Rigo, Carl Friedrich Bolz, Maciej Fijalkowski +Antonio Cuni, Samuele Pedroni, Niko Matsakis, +Holger Krekel, Alexander Schremmer, Amaury Forgeot d Arc, +Christian Tismer, Jacob Hallen, Laura Creighton, Beatrice During, +Leonardo Santagada, Stephan Diehl, Lene Wagner, +Jean-Paul Calderone, Michael Schneider, Lene Wagner, +Michael Hudson, Richard Emslie, Eric van Riet Paap, +Aurelien Campeas, Nikolaus Haldimann, ...

+
+
+

Involved Organisations

+

merlinux: Holger Krekel, Maciej Fijalkowski, Antonio Cuni, Carl Friedrich Bolz, Lene Wagner

+

Open End: Samuele Pedroni, Bea During, Jacob Hallen, Laura Creighton

+

Heinrich Heine University: Carl Friedrich Bolz, Michael Leuschel, (Armin Rigo)

+

Tismerysoft: Christian Tismer

+

University of Genova: Antonio Cuni

+
+
+

Contact, questions, discussion

+

Holger Krekel <holger@merlinux.de>

+

Beatrice During <bea@openend.se>

+

pypy-dev@codespeak.net +#pypy on irc.freenode.org

+

http://codespeak.net/pypy

+

http://merlinux.eu

+

http://openend.se

+
+
+ + From hpk at codespeak.net Mon Jul 14 09:44:43 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Mon, 14 Jul 2008 09:44:43 +0200 (CEST) Subject: [pypy-svn] r56531 - pypy/extradoc/talk/ep2008 Message-ID: <20080714074443.BCA5E169EE1@codespeak.net> Author: hpk Date: Mon Jul 14 09:44:43 2008 New Revision: 56531 Removed: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.html Log: removing it, does not help like this. From hpk at codespeak.net Mon Jul 14 15:22:41 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Mon, 14 Jul 2008 15:22:41 +0200 (CEST) Subject: [pypy-svn] r56538 - pypy/extradoc/talk/ep2008 Message-ID: <20080714132241.A9B1439B5AA@codespeak.net> Author: hpk Date: Mon Jul 14 15:22:40 2008 New Revision: 56538 Modified: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt Log: fix year number, thanks pedronis Modified: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt ============================================================================== --- pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt (original) +++ pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt Mon Jul 14 15:22:40 2008 @@ -19,7 +19,7 @@ The When of PyPy --------------------- -- 2001: startup of PyPy OSS work on codespeak +- 2003: startup of PyPy OSS work on codespeak (get your own JIT for your favourite language for free) - 2004-2007: consolidation through EU-project (results as well as growing community) From hpk at codespeak.net Mon Jul 14 15:27:10 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Mon, 14 Jul 2008 15:27:10 +0200 (CEST) Subject: [pypy-svn] r56539 - pypy/extradoc/talk/ep2008 Message-ID: <20080714132710.3813E39B592@codespeak.net> Author: hpk Date: Mon Jul 14 15:27:09 2008 New Revision: 56539 Modified: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt Log: fixing this slide some more now that we point to it from the blog Modified: pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt ============================================================================== --- pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt (original) +++ pypy/extradoc/talk/ep2008/ep2008-talk-behind-the-scenes.txt Mon Jul 14 15:27:09 2008 @@ -20,9 +20,9 @@ The When of PyPy --------------------- - 2003: startup of PyPy OSS work on codespeak - (get your own JIT for your favourite language for free) + (Python written in Python / Minimal Python) - 2004-2007: consolidation through EU-project - (results as well as growing community) + (get your own JIT for your favourite language for free) - 2008--> towards production ready through company funding Development culture From bgola at codespeak.net Mon Jul 14 18:31:04 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Mon, 14 Jul 2008 18:31:04 +0200 (CEST) Subject: [pypy-svn] r56540 - in pypy/branch/2.5-features/pypy/interpreter: . test Message-ID: <20080714163104.5817C16A08F@codespeak.net> Author: bgola Date: Mon Jul 14 18:31:01 2008 New Revision: 56540 Modified: pypy/branch/2.5-features/pypy/interpreter/generator.py pypy/branch/2.5-features/pypy/interpreter/pyframe.py pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py pypy/branch/2.5-features/pypy/interpreter/typedef.py Log: throw() and close() implemented, still not working because of the magic number, see execute_generator_frame() in pyframe.py. Modified: pypy/branch/2.5-features/pypy/interpreter/generator.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/generator.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/generator.py Mon Jul 14 18:31:01 2008 @@ -31,6 +31,9 @@ def descr_send(self, w_arg=None): """send(arg) -> send 'arg' into generator, return next yielded value or raise StopIteration.""" + return self.send_ex(w_arg) + + def send_ex(self, w_arg, exc=False): space = self.space if self.running: raise OperationError(space.w_ValueError, @@ -47,7 +50,7 @@ self.running = True try: try: - w_result = self.frame.execute_generator_frame(w_arg) + w_result = self.frame.execute_generator_frame(w_arg, exc) except OperationError: # errors finish a frame self.frame.frame_finished_execution = True @@ -61,8 +64,64 @@ self.frame.f_back = None self.running = False + def descr_throw(self, w_type, w_val=None, w_tb=None): + """throw(typ[,val[,tb]]) -> raise exception in generator, +return next yielded value or raise StopIteration.""" + from pypy.interpreter.typedef import PyTraceback + space = self.space + + if space.is_w(w_tb, space.w_None): + w_tb = None + + if w_val is None: + w_val = space.w_None + + if w_tb is not None: + if not space.is_true(space.isinstance(w_tb, + space.gettypeobject(PyTraceback.typedef))): + msg = "throw() third argument must be a traceback object" + raise OperationError(space.w_TypeError, space.wrap(msg)) + + if space.is_true(space.abstract_isclass(w_type)) and \ + space.is_true(space.issubtype(w_type, space.w_BaseException)): + exception = OperationError(w_type, w_val, w_tb) + + elif space.is_true(space.isinstance(w_type, space.w_BaseException)): + if not space.is_w(w_val, space.w_None): + msg = "instance exception may not have a separate value" + raise OperationError(space.w_TypeError, space.wrap(msg)) + else: + exception = OperationError(w_type.getclass(space), w_val, w_tb) + + else: + if not space.is_true(space.isinstance(w_type, space.w_str)): + msg = "exceptions must be classes, or instances, not %s" % ( + w_type.typedef.name) + raise OperationError(space.w_TypeError, space.wrap(msg)) + else: + exception = OperationError(w_type, w_val) + + ec = space.getexecutioncontext() + next_instr = self.frame.handle_operation_error(ec, exception) + self.frame.last_instr = next_instr - 1 + + return self.send_ex(space.w_None, True) + def descr_next(self): - """x.next() -> the next value, or raise StopIteration""" - return self.descr_send() + """next() -> the next value, or raise StopIteration""" + return self.send_ex(self.space.w_None) - + def descr_close(self): + """close(arg) -> raise GeneratorExit inside generator.""" + space = self.space + try: + w_retval = self.descr_throw(space.w_GeneratorExit) + except OperationError, e: + if e.match(space, space.w_StopIteration) or \ + e.match(space, space.w_GeneratorExit): + return space.w_None + raise + + if w_retval is not None or not space.is_w(w_retval, space.None): + msg = "generator ignored GeneratorExit" + raise OperationError(space.w_RuntimeError, space.wrap(msg)) Modified: pypy/branch/2.5-features/pypy/interpreter/pyframe.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyframe.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyframe.py Mon Jul 14 18:31:01 2008 @@ -19,7 +19,6 @@ g[op] = opcode.opmap[op] HAVE_ARGUMENT = opcode.HAVE_ARGUMENT - class PyFrame(eval.Frame): """Represents a frame for a regular Python function that needs to be interpreted. @@ -92,11 +91,11 @@ else: return self.execute_frame() - def execute_generator_frame(self, w_inputvalue): + def execute_generator_frame(self, w_inputvalue, ex=False): # opcode semantic change in CPython 2.5: we must pass an input value # when resuming a generator, which goes into the value stack. - # (it's always w_None for now - not implemented in generator.py) - if self.pycode.magic >= 0xa0df294 and self.last_instr != -1: + # It's not working because the value of magic must be changed in PyCode + if self.pycode.magic >= 0xa0df294 and self.last_instr != -1 and not ex: self.pushvalue(w_inputvalue) return self.execute_frame() Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/test/test_generator.py Mon Jul 14 18:31:01 2008 @@ -34,6 +34,113 @@ g.next() assert g.send(42) == 42 + def test_throw1(self): + def f(): + yield 2 + g = f() + raises(NameError, g.throw, NameError, "Error") + + def test_throw2(self): + def f(): + yield 2 + g = f() + raises(NameError, g.throw, NameError("Error")) + + def test_throw3(self): + def f(): + try: + yield 1 + yield 2 + except: + yield 3 + g = f() + assert g.next() == 1 + assert g.throw(NameError("Error")) == 3 + raises(StopIteration, g.next) + + def test_throw3(self): + def f(): + try: + yield 1 + v = (yield 2) + except: + yield 3 + g = f() + assert g.next() == 1 + assert g.next() == 2 + assert g.throw(NameError("Error")) == 3 + raises(StopIteration, g.next) + + def test_throw4(self): + def f(): + try: + yield 1 + except: + x = 3 + try: + yield x + except: + pass + g = f() + g.next() + # String exceptions are allowed (with DeprecationWarning) + assert g.throw("Error") == 3 + raises(StopIteration, g.throw, "Error") + + def test_throw_fail(self): + def f(): + yield 1 + g = f() + raises(TypeError, g.throw, NameError("Error"), "error") + + def test_throw_fail2(self): + def f(): + yield 1 + g = f() + raises(TypeError, g.throw, list()) + + def test_throw_fail3(self): + def f(): + yield 1 + g = f() + raises(TypeError, g.throw, NameError("Error"), None, "not tb object") + + def test_close(self): + def f(): + yield 1 + g = f() + assert g.close() is None + + def test_close2(self): + def f(): + try: + yield 1 + except GeneratorExit: + raise StopIteration + g = f() + g.next() + assert g.close() is None + + def test_close3(self): + def f(): + try: + yield 1 + except GeneratorExit: + raise NameError + g = f() + g.next() + raises(NameError, g.close) + + def test_close_fail(self): + def f(): + try: + yield 1 + except GeneratorExit: + yield 2 + g = f() + g.next() + raises(RuntimeError, g.close) + def test_generator_raises_typeerror(self): def f(): yield 1 Modified: pypy/branch/2.5-features/pypy/interpreter/typedef.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/typedef.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/typedef.py Mon Jul 14 18:31:01 2008 @@ -819,6 +819,10 @@ descrmismatch='next'), send = interp2app(GeneratorIterator.descr_send, descrmismatch='send'), + throw = interp2app(GeneratorIterator.descr_throw, + descrmismatch='throw'), + close = interp2app(GeneratorIterator.descr_close, + descrmismatch='close'), __iter__ = interp2app(GeneratorIterator.descr__iter__, descrmismatch='__iter__'), gi_running = interp_attrproperty('running', cls=GeneratorIterator), From arigo at codespeak.net Wed Jul 16 01:19:35 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 16 Jul 2008 01:19:35 +0200 (CEST) Subject: [pypy-svn] r56584 - pypy/dist/pypy/doc/config Message-ID: <20080715231935.8FC292A80A0@codespeak.net> Author: arigo Date: Wed Jul 16 01:19:33 2008 New Revision: 56584 Added: pypy/dist/pypy/doc/config/objspace.lonepycfiles.txt (contents, props changed) Modified: pypy/dist/pypy/doc/config/objspace.usepycfiles.txt Log: Document the pyc files options. Added: pypy/dist/pypy/doc/config/objspace.lonepycfiles.txt ============================================================================== --- (empty file) +++ pypy/dist/pypy/doc/config/objspace.lonepycfiles.txt Wed Jul 16 01:19:33 2008 @@ -0,0 +1,16 @@ +If turned on, PyPy accepts to import a module ``x`` if it finds a +file ``x.pyc`` even if there is no file ``x.py``. + +This is the way that CPython behaves, but it is disabled by +default for PyPy because it is a common cause of issues: most +typically, the ``x.py`` file is removed (manually or by a +version control system) but the ``x`` module remains +accidentally importable because the ``x.pyc`` file stays +around. + +The usual reason for wanting this feature is to distribute +non-open-source Python programs by distributing ``pyc`` files +only, but this use case is not practical for PyPy at the +moment because multiple versions of PyPy compiled with various +optimizations might be unable to load each other's ``pyc`` +files. Modified: pypy/dist/pypy/doc/config/objspace.usepycfiles.txt ============================================================================== --- pypy/dist/pypy/doc/config/objspace.usepycfiles.txt (original) +++ pypy/dist/pypy/doc/config/objspace.usepycfiles.txt Wed Jul 16 01:19:33 2008 @@ -1,4 +1,4 @@ If this option is used, then PyPy imports and generates "pyc" files in the -usual way. This is mostly always on, -except when using other options that lead to PyPy-specific bytecodes that should -not be cached on disk because they might confuse CPython. +same way as CPython. This is true by default and there is not much reason +to turn it off nowadays. If off, PyPy never produces "pyc" files and +ignores any "pyc" file that might already be present. From hpk at codespeak.net Wed Jul 16 10:27:36 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Wed, 16 Jul 2008 10:27:36 +0200 (CEST) Subject: [pypy-svn] r56587 - pypy/dist/pypy/translator/c Message-ID: <20080716082736.D8E55169E83@codespeak.net> Author: hpk Date: Wed Jul 16 10:27:34 2008 New Revision: 56587 Modified: pypy/dist/pypy/translator/c/genc.py Log: refactor ld/cflags options a bit to make a simple pypy-c build work on solaris/SunOS 5.10 (i.e. no-threads and no-allworkingmodules) Modified: pypy/dist/pypy/translator/c/genc.py ============================================================================== --- pypy/dist/pypy/translator/c/genc.py (original) +++ pypy/dist/pypy/translator/c/genc.py Wed Jul 16 10:27:34 2008 @@ -341,12 +341,7 @@ else: eci = self.eci.merge(ExternalCompilationInfo(includes= [str(self.targetdir)])) - if sys.platform == 'darwin': - compiler.compile_extra.append('-mdynamic-no-pic') - if self.config.translation.compilerflags: - compiler.compile_extra.append(self.config.translation.compilerflags) - if self.config.translation.linkerflags: - compiler.link_extra.append(self.config.translation.linkerflags) + self.adaptflags(compiler) compiler.build() self.executable_name = str(compiler.outputfilename) self._compiled = True @@ -356,6 +351,16 @@ assert self._compiled return py.process.cmdexec('"%s" %s' % (self.executable_name, args)) + def adaptflags(self, compiler): + if sys.platform == 'darwin': + compiler.compile_extra.append('-mdynamic-no-pic') + if sys.platform == 'sunos5': + compiler.link_extra.append("-lrt") + if self.config.translation.compilerflags: + compiler.compile_extra.append(self.config.translation.compilerflags) + if self.config.translation.linkerflags: + compiler.link_extra.append(self.config.translation.linkerflags) + def gen_makefile(self, targetdir): def write_list(lst, prefix): for i, fn in enumerate(lst): @@ -369,12 +374,8 @@ self.eci = self.eci.merge(ExternalCompilationInfo( includes=['.', str(self.targetdir)])) compiler = self.getccompiler() - if sys.platform == 'darwin': - compiler.compile_extra.append('-mdynamic-no-pic') - if self.config.translation.compilerflags: - compiler.compile_extra.append(self.config.translation.compilerflags) - if self.config.translation.linkerflags: - compiler.link_extra.append(self.config.translation.linkerflags) + + self.adaptflags(compiler) assert self.config.translation.gcrootfinder != "llvmgc" cfiles = [] ofiles = [] From bgola at codespeak.net Wed Jul 16 15:18:44 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Wed, 16 Jul 2008 15:18:44 +0200 (CEST) Subject: [pypy-svn] r56588 - pypy/branch/2.5-features/pypy/lib Message-ID: <20080716131844.1E663169ED1@codespeak.net> Author: bgola Date: Wed Jul 16 15:18:42 2008 New Revision: 56588 Modified: pypy/branch/2.5-features/pypy/lib/_exceptions.py Log: GeneratorExit exception Modified: pypy/branch/2.5-features/pypy/lib/_exceptions.py ============================================================================== --- pypy/branch/2.5-features/pypy/lib/_exceptions.py (original) +++ pypy/branch/2.5-features/pypy/lib/_exceptions.py Wed Jul 16 15:18:42 2008 @@ -103,6 +103,9 @@ class Exception(BaseException): """Common base class for all non-exit exceptions.""" +class GeneratorExit(Exception): + """Request that a generator exit.""" + class StandardError(Exception): """Base class for all standard Python exceptions.""" From bgola at codespeak.net Wed Jul 16 15:23:26 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Wed, 16 Jul 2008 15:23:26 +0200 (CEST) Subject: [pypy-svn] r56589 - in pypy/branch/2.5-features/pypy: interpreter interpreter/astcompiler module/__builtin__ Message-ID: <20080716132326.9CCE1169ED1@codespeak.net> Author: bgola Date: Wed Jul 16 15:23:25 2008 New Revision: 56589 Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py pypy/branch/2.5-features/pypy/interpreter/pycode.py pypy/branch/2.5-features/pypy/module/__builtin__/importing.py Log: new magic number for .pyc files and for pycode objects (they are different numbers). The one for PyCode object is the same for CPython 2.5c2 and is checked through the code to keep compatibility against other bytecode versions. As we are chaning some opcodes (as YIELD_VALUE and IMPORT_NAME) the .pyc magic number have to be changed as well because the old bytecode generated by pypy is not compatible anymore and we need to recompile the .py files Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/astcompiler/pyassem.py Wed Jul 16 15:23:25 2008 @@ -534,7 +534,7 @@ 'STORE_FAST': -1, 'DELETE_FAST': 0, 'IMPORT_STAR': -1, - 'IMPORT_NAME': 0, + 'IMPORT_NAME': -1, 'IMPORT_FROM': 1, 'LOAD_ATTR': 0, # unlike other loads 'GET_ITER': 0, Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py Wed Jul 16 15:23:25 2008 @@ -866,6 +866,7 @@ def visitImport(self, node): self.set_lineno(node) for name, alias in node.names: + self.emitop_obj('LOAD_CONST', self.space.wrap(-1)) # 2.5 flag self.emitop_obj('LOAD_CONST', self.space.w_None) self.emitop('IMPORT_NAME', name) mod = name.split(".")[0] @@ -878,6 +879,7 @@ def visitFrom(self, node): self.set_lineno(node) fromlist = [ self.space.wrap(name) for name,alias in node.names ] + self.emitop_obj('LOAD_CONST', self.space.wrap(-1)) # 2.5 flag self.emitop_obj('LOAD_CONST', self.space.newtuple(fromlist)) self.emitop('IMPORT_NAME', node.modname) for name, alias in node.names: Modified: pypy/branch/2.5-features/pypy/interpreter/pycode.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pycode.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/pycode.py Wed Jul 16 15:23:25 2008 @@ -56,7 +56,7 @@ def __init__(self, space, argcount, nlocals, stacksize, flags, code, consts, names, varnames, filename, name, firstlineno, lnotab, freevars, cellvars, - hidden_applevel=False, magic = 62061 | 0x0a0d0000): # value for Python 2.4.1 + hidden_applevel=False, magic = 62131 | 0x0a0d0000): # value for Python 2.5c2 """Initialize a new code object from parameters given by the pypy compiler""" self.space = space Modified: pypy/branch/2.5-features/pypy/module/__builtin__/importing.py ============================================================================== --- pypy/branch/2.5-features/pypy/module/__builtin__/importing.py (original) +++ pypy/branch/2.5-features/pypy/module/__builtin__/importing.py Wed Jul 16 15:23:25 2008 @@ -395,8 +395,8 @@ # * CALL_LIKELY_BUILTIN +2 # * CALL_METHOD +4 # -# -MAGIC = 1024 | (ord('\r')<<16) | (ord('\n')<<24) +# +MAGIC = 1034 | (ord('\r')<<16) | (ord('\n')<<24) MARSHAL_VERSION_FOR_PYC = 2 def get_pyc_magic(space): From bgola at codespeak.net Wed Jul 16 15:38:23 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Wed, 16 Jul 2008 15:38:23 +0200 (CEST) Subject: [pypy-svn] r56591 - pypy/branch/2.5-features/pypy/interpreter Message-ID: <20080716133823.AB99C169F42@codespeak.net> Author: bgola Date: Wed Jul 16 15:38:23 2008 New Revision: 56591 Modified: pypy/branch/2.5-features/pypy/interpreter/generator.py Log: refactoring throw and close methods Modified: pypy/branch/2.5-features/pypy/interpreter/generator.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/generator.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/generator.py Wed Jul 16 15:38:23 2008 @@ -67,9 +67,13 @@ def descr_throw(self, w_type, w_val=None, w_tb=None): """throw(typ[,val[,tb]]) -> raise exception in generator, return next yielded value or raise StopIteration.""" + return self.throw(w_type, w_val, w_tb) + + + def throw(self, w_type, w_val=None, w_tb=None): from pypy.interpreter.typedef import PyTraceback space = self.space - + if space.is_w(w_tb, space.w_None): w_tb = None @@ -99,7 +103,7 @@ w_type.typedef.name) raise OperationError(space.w_TypeError, space.wrap(msg)) else: - exception = OperationError(w_type, w_val) + exception = OperationError(w_type, w_val, w_tb) ec = space.getexecutioncontext() next_instr = self.frame.handle_operation_error(ec, exception) @@ -115,7 +119,7 @@ """close(arg) -> raise GeneratorExit inside generator.""" space = self.space try: - w_retval = self.descr_throw(space.w_GeneratorExit) + w_retval = self.throw(space.w_GeneratorExit) except OperationError, e: if e.match(space, space.w_StopIteration) or \ e.match(space, space.w_GeneratorExit): From cfbolz at codespeak.net Wed Jul 16 16:15:31 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 16 Jul 2008 16:15:31 +0200 (CEST) Subject: [pypy-svn] r56592 - pypy/branch/2.5-features/pypy/interpreter Message-ID: <20080716141531.ECA4F2A01B6@codespeak.net> Author: cfbolz Date: Wed Jul 16 16:15:29 2008 New Revision: 56592 Modified: pypy/branch/2.5-features/pypy/interpreter/generator.py Log: A small tweak: use NoneNotWrapped as a default value to get real Nones. Modified: pypy/branch/2.5-features/pypy/interpreter/generator.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/generator.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/generator.py Wed Jul 16 16:15:29 2008 @@ -1,5 +1,6 @@ from pypy.interpreter.error import OperationError from pypy.interpreter.baseobjspace import Wrappable +from pypy.interpreter.gateway import NoneNotWrapped class GeneratorIterator(Wrappable): @@ -64,22 +65,16 @@ self.frame.f_back = None self.running = False - def descr_throw(self, w_type, w_val=None, w_tb=None): + def descr_throw(self, w_type, w_val=None, w_tb=NoneNotWrapped): """throw(typ[,val[,tb]]) -> raise exception in generator, return next yielded value or raise StopIteration.""" return self.throw(w_type, w_val, w_tb) - def throw(self, w_type, w_val=None, w_tb=None): + def throw(self, w_type, w_val, w_tb): from pypy.interpreter.typedef import PyTraceback space = self.space - if space.is_w(w_tb, space.w_None): - w_tb = None - - if w_val is None: - w_val = space.w_None - if w_tb is not None: if not space.is_true(space.isinstance(w_tb, space.gettypeobject(PyTraceback.typedef))): @@ -119,7 +114,7 @@ """close(arg) -> raise GeneratorExit inside generator.""" space = self.space try: - w_retval = self.throw(space.w_GeneratorExit) + w_retval = self.throw(space.w_GeneratorExit, space.w_None, None) except OperationError, e: if e.match(space, space.w_StopIteration) or \ e.match(space, space.w_GeneratorExit): From bgola at codespeak.net Wed Jul 16 17:18:02 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Wed, 16 Jul 2008 17:18:02 +0200 (CEST) Subject: [pypy-svn] r56597 - pypy/branch/2.5-features/pypy/config Message-ID: <20080716151802.298AE398008@codespeak.net> Author: bgola Date: Wed Jul 16 17:18:01 2008 New Revision: 56597 Modified: pypy/branch/2.5-features/pypy/config/pypyoption.py Log: chaning grammar default version to 2.5 Modified: pypy/branch/2.5-features/pypy/config/pypyoption.py ============================================================================== --- pypy/branch/2.5-features/pypy/config/pypyoption.py (original) +++ pypy/branch/2.5-features/pypy/config/pypyoption.py Wed Jul 16 17:18:01 2008 @@ -87,7 +87,7 @@ cmdline='--compiler'), ChoiceOption("pyversion", "which grammar to use for app-level code", - ["2.3", "2.4", "2.5"], "2.4", + ["2.3", "2.4", "2.5"], "2.5", cmdline='--pyversion'), OptionDescription("opcodes", "opcodes to enable in the interpreter", [ From exarkun at codespeak.net Wed Jul 16 19:27:09 2008 From: exarkun at codespeak.net (exarkun at codespeak.net) Date: Wed, 16 Jul 2008 19:27:09 +0200 (CEST) Subject: [pypy-svn] r56598 - pypy/build/bot Message-ID: <20080716172709.21E6F169F95@codespeak.net> Author: exarkun Date: Wed Jul 16 19:27:08 2008 New Revision: 56598 Modified: pypy/build/bot/pypybuilders.py Log: re-instate uploads; but disable testing, because the reporter doesn't work at all Modified: pypy/build/bot/pypybuilders.py ============================================================================== --- pypy/build/bot/pypybuilders.py (original) +++ pypy/build/bot/pypybuilders.py Wed Jul 16 19:27:08 2008 @@ -101,6 +101,8 @@ class PyPyBuildFactory(BuildFactory): PyTestStep = None + executable = "pypy-c" + def __init__(self, pytestArguments, translationArguments, targetArguments, *a, **kw): BuildFactory.__init__(self, *a, **kw) @@ -136,32 +138,33 @@ translationArgs=translationArguments, targetArgs=targetArguments) - if pytestArguments is not None: - self.addStep( - self.PyTestStep, - testPackage="pypy", - testArguments=pytestArguments, - timeout=60 * 60) - - self.addStep( - self.PyTestStep, - python="pypy/translator/goal/pypy-c", - testPackage="pypy", - # Exclude --boxed if it was specified, because -A shouldn't - # really use much memory. - testArguments=["-A"] + list(arg for arg in pytestArguments if arg != "--boxed"), - timeout=60 * 60) - -# self.addStep(FileUpload, -# slavesrc="pypy-src/pypy/translator/goal/pypy-c", -# masterdest="pypy-c") -# + uploadSuffix = "".join(translationArguments + targetArguments) + self.addStep(FileUpload, + slavesrc="pypy-src/pypy/translator/goal/" + self.executable, + masterdest="public_html/builds/pypy-c-" + uploadSuffix) + +# if pytestArguments is not None: +# self.addStep( +# self.PyTestStep, +# testPackage="pypy", +# testArguments=pytestArguments, +# timeout=60 * 60) + +# self.addStep( +# self.PyTestStep, +# python="pypy/translator/goal/pypy-c", +# testPackage="pypy", +# # Exclude --boxed if it was specified, because -A shouldn't +# # really use much memory. +# testArguments=["-A"] + list(arg for arg in pytestArguments if arg != "--boxed"), +# timeout=60 * 60) + # self.addStep( # SVN, # workdir="build/Twisted-src", # svnurl="svn://svn.twistedmatrix.com/svn/Twisted/trunk", # mode="copy") -# + # self.addStep( # Trial, # workdir="build/pypy-src/pypy/translator/goal", @@ -181,3 +184,4 @@ class WindowsPyPyBuildFactory(PyPyBuildFactory): PyTestStep = WindowsPyTest + executable = "pypy-c.exe" From exarkun at codespeak.net Wed Jul 16 20:09:27 2008 From: exarkun at codespeak.net (exarkun at codespeak.net) Date: Wed, 16 Jul 2008 20:09:27 +0200 (CEST) Subject: [pypy-svn] r56599 - in pypy/dist/pypy/rlib: . test Message-ID: <20080716180927.64DFB169F81@codespeak.net> Author: exarkun Date: Wed Jul 16 20:09:25 2008 New Revision: 56599 Modified: pypy/dist/pypy/rlib/rzipfile.py pypy/dist/pypy/rlib/test/test_rzipfile.py Log: Apply a 32bit mask to the CRC32 value to make the check correct on 64bit platforms Modified: pypy/dist/pypy/rlib/rzipfile.py ============================================================================== --- pypy/dist/pypy/rlib/rzipfile.py (original) +++ pypy/dist/pypy/rlib/rzipfile.py Wed Jul 16 20:09:25 2008 @@ -181,7 +181,8 @@ # file_offset must be computed below... (x.create_version, x.create_system, x.extract_version, x.reserved, x.flag_bits, x.compress_type, t, d, - x.CRC, x.compress_size, x.file_size) = centdir[1:12] + crc, x.compress_size, x.file_size) = centdir[1:12] + x.CRC = r_uint(crc) & 0xffffffff x.dostime = t x.dosdate = d x.volume, x.internal_attr, x.external_attr = centdir[15:18] @@ -237,7 +238,7 @@ "Unsupported compression method %d for file %s" % \ (zinfo.compress_type, filename) crc = crc32(bytes) - if crc != r_uint(zinfo.CRC): + if crc != zinfo.CRC: raise BadZipfile, "Bad CRC-32 for file %s" % filename return bytes Modified: pypy/dist/pypy/rlib/test/test_rzipfile.py ============================================================================== --- pypy/dist/pypy/rlib/test/test_rzipfile.py (original) +++ pypy/dist/pypy/rlib/test/test_rzipfile.py Wed Jul 16 20:09:25 2008 @@ -16,6 +16,10 @@ cls.year = time.localtime(time.time())[0] zipfile.writestr("one", "stuff") zipfile.writestr("dir" + os.path.sep + "two", "otherstuff") + # Value selected to produce a CRC32 which is negative if + # interpreted as a signed 32 bit integer. This exercises the + # masking behavior necessary on 64 bit platforms. + zipfile.writestr("three", "hello, world") zipfile.close() def test_rzipfile(self): @@ -26,7 +30,8 @@ rzip = RZipFile(zipname, "r", compression) info = rzip.getinfo('one') return (info.date_time[0] == year and - rzip.read('one') == 'stuff') + rzip.read('one') == 'stuff' and + rzip.read('three') == 'hello, world') assert one() assert self.interpret(one, []) From exarkun at codespeak.net Wed Jul 16 20:22:36 2008 From: exarkun at codespeak.net (exarkun at codespeak.net) Date: Wed, 16 Jul 2008 20:22:36 +0200 (CEST) Subject: [pypy-svn] r56600 - pypy/dist/pypy/lib Message-ID: <20080716182236.065F2169FB2@codespeak.net> Author: exarkun Date: Wed Jul 16 20:22:36 2008 New Revision: 56600 Modified: pypy/dist/pypy/lib/imp.py Log: Use "i" instead of "L" so that the magic string comes out as 4 bytes, even on 64bit platforms Modified: pypy/dist/pypy/lib/imp.py ============================================================================== --- pypy/dist/pypy/lib/imp.py (original) +++ pypy/dist/pypy/lib/imp.py Wed Jul 16 20:22:36 2008 @@ -25,7 +25,7 @@ def get_magic(): """Return the magic number for .pyc or .pyo files.""" import struct - return struct.pack('L', _get_magic_as_int()) + return struct.pack(' Author: arigo Date: Wed Jul 16 22:52:23 2008 New Revision: 56601 Modified: pypy/dist/pypy/lib/_pypy_interact.py Log: And now for something completely different Modified: pypy/dist/pypy/lib/_pypy_interact.py ============================================================================== --- pypy/dist/pypy/lib/_pypy_interact.py (original) +++ pypy/dist/pypy/lib/_pypy_interact.py Wed Jul 16 22:52:23 2008 @@ -5,8 +5,14 @@ def interactive_console(mainmodule=None): try: - import _pypy_irc_topic - print "``%s''" % (_pypy_irc_topic.some_topic(),) + from _pypy_irc_topic import some_topic + text = "And now for something completely different: ``%s''" % ( + some_topic(),) + if len(text) >= 80: + i = text[:80].rfind(' ') + print text[:i] + text = text[i+1:] + print text except ImportError: pass try: From arigo at codespeak.net Wed Jul 16 23:16:46 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 16 Jul 2008 23:16:46 +0200 (CEST) Subject: [pypy-svn] r56602 - pypy/dist/pypy/rpython/module Message-ID: <20080716211646.6AA562A807A@codespeak.net> Author: arigo Date: Wed Jul 16 23:16:45 2008 New Revision: 56602 Modified: pypy/dist/pypy/rpython/module/ll_os.py Log: Extend the hackish workaround for the gcc -profile bug on OS X for the fork() function to apply it to execv() and execve() too. Modified: pypy/dist/pypy/rpython/module/ll_os.py ============================================================================== --- pypy/dist/pypy/rpython/module/ll_os.py (original) +++ pypy/dist/pypy/rpython/module/ll_os.py Wed Jul 16 23:16:45 2008 @@ -133,8 +133,12 @@ @registering_if(os, 'execv') def register_os_execv(self): - os_execv = self.llexternal('execv', [rffi.CCHARP, rffi.CCHARPP], - rffi.INT) + eci = self.gcc_profiling_bug_workaround( + 'int _noprof_execv(char *path, char *argv[])', + 'return execv(path, argv);') + os_execv = self.llexternal('_noprof_execv', + [rffi.CCHARP, rffi.CCHARPP], + rffi.INT, compilation_info = eci) def execv_llimpl(path, args): l_args = rffi.liststr2charpp(args) @@ -148,8 +152,12 @@ @registering_if(os, 'execve') def register_os_execve(self): + eci = self.gcc_profiling_bug_workaround( + 'int _noprof_execve(char *filename, char *argv[], char *envp[])', + 'return execve(filename, argv, envp);') os_execve = self.llexternal( - 'execve', [rffi.CCHARP, rffi.CCHARPP, rffi.CCHARPP], rffi.INT) + '_noprof_execve', [rffi.CCHARP, rffi.CCHARPP, rffi.CCHARPP], + rffi.INT, compilation_info = eci) def execve_llimpl(path, args, env): # XXX Check path, args, env for \0 and raise TypeErrors as @@ -1121,16 +1129,8 @@ @registering_if(os, 'fork') def register_os_fork(self): - # XXX horrible workaround for a bug of profiling in gcc on - # OS X with functions containing a direct call to fork() - eci = ExternalCompilationInfo( - post_include_bits = ['pid_t _noprof_fork(void);'], - separate_module_sources = [''' - /*--no-profiling-for-this-file!--*/ - pid_t _noprof_fork(void) { - return fork(); - } - ''']) + eci = self.gcc_profiling_bug_workaround('pid_t _noprof_fork(void)', + 'return fork();') os_fork = self.llexternal('_noprof_fork', [], rffi.PID_T, compilation_info = eci) @@ -1222,6 +1222,19 @@ return extdef([int], str, "ll_os.ttyname", llimpl=ttyname_llimpl) + # ____________________________________________________________ + # XXX horrible workaround for a bug of profiling in gcc on + # OS X with functions containing a direct call to some system calls + # like fork(), execv(), execve() + def gcc_profiling_bug_workaround(self, decl, body): + body = ('/*--no-profiling-for-this-file!--*/\n' + '%s {\n' + '\t%s\n' + '}\n' % (decl, body,)) + return ExternalCompilationInfo( + post_include_bits = [decl + ';'], + separate_module_sources = [body]) + # ____________________________________________________________ # Support for os.environ From arigo at codespeak.net Wed Jul 16 23:30:37 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 16 Jul 2008 23:30:37 +0200 (CEST) Subject: [pypy-svn] r56603 - pypy/dist/pypy/lib Message-ID: <20080716213037.EFA0E2A80A0@codespeak.net> Author: arigo Date: Wed Jul 16 23:30:37 2008 New Revision: 56603 Modified: pypy/dist/pypy/lib/_hashlib.py Log: Untabbify. Modified: pypy/dist/pypy/lib/_hashlib.py ============================================================================== --- pypy/dist/pypy/lib/_hashlib.py (original) +++ pypy/dist/pypy/lib/_hashlib.py Wed Jul 16 23:30:37 2008 @@ -8,153 +8,153 @@ # FIXME do we really need this anywhere here? class ENV_MD(Structure): - # XXX Should there be more to this object?. - _fields_ = [ - ('type', c_int), - ('pkey_type', c_int), - ('md_size', c_int), - ] + # XXX Should there be more to this object?. + _fields_ = [ + ('type', c_int), + ('pkey_type', c_int), + ('md_size', c_int), + ] class _dummy_env_md(Structure): - # XXX used for OS X, a bit hairy - _fields_ = [ - ('digest', ENV_MD), - ('two', c_int), - ('three', c_int), - ('four', c_int), - ('five', c_int), - ] + # XXX used for OS X, a bit hairy + _fields_ = [ + ('digest', ENV_MD), + ('two', c_int), + ('three', c_int), + ('four', c_int), + ('five', c_int), + ] def _new_ENV_MD(): - return _dummy_env_md() + return _dummy_env_md() # OpenSSL initialization lib.OpenSSL_add_all_digests() def _get_digest(ctx): - return ctx.digest + return ctx.digest # taken from evp.h, max size is 512 bit, 64 chars lib.EVP_MAX_MD_SIZE = 64 class hash(object): - """ - A hash represents the object used to calculate a checksum of a - string of information. - - Methods: - - update() -- updates the current digest with an additional string - digest() -- return the current digest value - hexdigest() -- return the current digest as a string of hexadecimal digits - copy() -- return a copy of the current hash object - - Attributes: - - name -- the hash algorithm being used by this object - digest_size -- number of bytes in this hashes output - """ - def __init__(self, obj, name): - self.name = name # part of API - #print 'obj is ', obj - if isinstance(obj, _dummy_env_md): - self._obj = obj.digest - else: - self._obj = obj # private - - def __repr__(self): - # format is not the same as in C module - return "<%s HASH object>" % (self.name) - - def copy(self): - "Return a copy of the hash object." - ctxnew = _new_ENV_MD() - lib.EVP_MD_CTX_copy(byref(ctxnew), byref(self._obj)) - return hash(ctxnew, self.name) - - def hexdigest(self): - "Return the digest value as a string of hexadecimal digits." - dig = self.digest() - a = [] - for x in dig: - a.append('%.2x' % ord(x)) - #print '\n--- %r \n' % ''.join(a) - return ''.join(a) - - def digest(self): - "Return the digest value as a string of binary data." - tmpctx = self.copy() - digest_size = tmpctx.digest_size - dig = create_string_buffer(lib.EVP_MAX_MD_SIZE) - lib.EVP_DigestFinal(byref(tmpctx._obj), dig, None) - lib.EVP_MD_CTX_cleanup(byref(tmpctx._obj)) - return dig.raw[:digest_size] - - def digest_size(self): - # XXX This isn't the nicest way, but the EVP_MD_size OpenSSL function - # XXX is defined as a C macro on OS X and would be significantly - # XXX harder to implement in another way. - # Values are digest sizes in bytes - return { - 'md5': 16, - 'sha1': 20, - 'sha224': 28, - 'sha256': 32, - 'sha384': 48, - 'sha512': 64, - }.get(self.name, 0) - digest_size = property(digest_size, None, None) # PEP 247 - digestsize = digest_size # deprecated, was once defined by sha module - - def block_size(self): - return lib.EVP_MD_CTX_block_size(byref(self._obj)) - block_size = property(block_size, None, None) - - def update(self, string): - "Update this hash object's state with the provided string." - lib.EVP_DigestUpdate(byref(self._obj), c_char_p(string), c_uint(len(string))) + """ + A hash represents the object used to calculate a checksum of a + string of information. + + Methods: + + update() -- updates the current digest with an additional string + digest() -- return the current digest value + hexdigest() -- return the current digest as a string of hexadecimal digits + copy() -- return a copy of the current hash object + + Attributes: + + name -- the hash algorithm being used by this object + digest_size -- number of bytes in this hashes output + """ + def __init__(self, obj, name): + self.name = name # part of API + #print 'obj is ', obj + if isinstance(obj, _dummy_env_md): + self._obj = obj.digest + else: + self._obj = obj # private + + def __repr__(self): + # format is not the same as in C module + return "<%s HASH object>" % (self.name) + + def copy(self): + "Return a copy of the hash object." + ctxnew = _new_ENV_MD() + lib.EVP_MD_CTX_copy(byref(ctxnew), byref(self._obj)) + return hash(ctxnew, self.name) + + def hexdigest(self): + "Return the digest value as a string of hexadecimal digits." + dig = self.digest() + a = [] + for x in dig: + a.append('%.2x' % ord(x)) + #print '\n--- %r \n' % ''.join(a) + return ''.join(a) + + def digest(self): + "Return the digest value as a string of binary data." + tmpctx = self.copy() + digest_size = tmpctx.digest_size + dig = create_string_buffer(lib.EVP_MAX_MD_SIZE) + lib.EVP_DigestFinal(byref(tmpctx._obj), dig, None) + lib.EVP_MD_CTX_cleanup(byref(tmpctx._obj)) + return dig.raw[:digest_size] + + def digest_size(self): + # XXX This isn't the nicest way, but the EVP_MD_size OpenSSL function + # XXX is defined as a C macro on OS X and would be significantly + # XXX harder to implement in another way. + # Values are digest sizes in bytes + return { + 'md5': 16, + 'sha1': 20, + 'sha224': 28, + 'sha256': 32, + 'sha384': 48, + 'sha512': 64, + }.get(self.name, 0) + digest_size = property(digest_size, None, None) # PEP 247 + digestsize = digest_size # deprecated, was once defined by sha module + + def block_size(self): + return lib.EVP_MD_CTX_block_size(byref(self._obj)) + block_size = property(block_size, None, None) + + def update(self, string): + "Update this hash object's state with the provided string." + lib.EVP_DigestUpdate(byref(self._obj), c_char_p(string), c_uint(len(string))) def new(name, string=''): - """ - Return a new hash object using the named algorithm. - An optional string argument may be provided and will be - automatically hashed. - - The MD5 and SHA1 algorithms are always supported. - """ - digest = lib.EVP_get_digestbyname(c_char_p(name)) - - if not isinstance(name, str): - raise TypeError("name must be a string") - if not digest: - raise ValueError("unknown hash function") - - ctx = _new_ENV_MD() - lib.EVP_DigestInit(pointer(ctx), digest) - - h = hash(_get_digest(ctx), name) - if string: - if not isinstance(string, str): - raise ValueError("hash content is not string") - h.update(string) - return hash(ctx, name) + """ + Return a new hash object using the named algorithm. + An optional string argument may be provided and will be + automatically hashed. + + The MD5 and SHA1 algorithms are always supported. + """ + digest = lib.EVP_get_digestbyname(c_char_p(name)) + + if not isinstance(name, str): + raise TypeError("name must be a string") + if not digest: + raise ValueError("unknown hash function") + + ctx = _new_ENV_MD() + lib.EVP_DigestInit(pointer(ctx), digest) + + h = hash(_get_digest(ctx), name) + if string: + if not isinstance(string, str): + raise ValueError("hash content is not string") + h.update(string) + return hash(ctx, name) # shortcut functions def openssl_md5(string=''): - return new('md5', string) + return new('md5', string) def openssl_sha1(string=''): - return new('sha1', string) + return new('sha1', string) def openssl_sha224(string=''): - return new('sha224', string) + return new('sha224', string) def openssl_sha256(string=''): - return new('sha256', string) + return new('sha256', string) def openssl_sha384(string=''): - return new('sha384', string) + return new('sha384', string) def openssl_sha512(string=''): - return new('sha512', string) + return new('sha512', string) From arigo at codespeak.net Wed Jul 16 23:36:30 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 16 Jul 2008 23:36:30 +0200 (CEST) Subject: [pypy-svn] r56604 - pypy/dist/pypy/lib/distributed/test Message-ID: <20080716213630.AEBD42A809A@codespeak.net> Author: arigo Date: Wed Jul 16 23:36:30 2008 New Revision: 56604 Modified: pypy/dist/pypy/lib/distributed/test/test_distributed.py Log: Fix test, broken when old-style classes became the default. Modified: pypy/dist/pypy/lib/distributed/test/test_distributed.py ============================================================================== --- pypy/dist/pypy/lib/distributed/test/test_distributed.py (original) +++ pypy/dist/pypy/lib/distributed/test/test_distributed.py Wed Jul 16 23:36:30 2008 @@ -78,7 +78,7 @@ assert fun(2, 3) == 5 def test_local_obj(self): - class A: + class A(object): def __init__(self, x): self.x = x @@ -137,7 +137,7 @@ assert d == xd def test_remote_obj(self): - class A: + class A(object): def __init__(self, x): self.x = x @@ -151,7 +151,7 @@ assert len(xa) == 11 def test_remote_doc_and_callback(self): - class A: + class A(object): """xxx""" def __init__(self): pass @@ -170,14 +170,14 @@ assert xa.meth(x) == 4 def test_double_reference(self): - class A: + class A(object): def meth(self, one): self.one = one def perform(self): return 1 + len(self.one()) - class B: + class B(object): def __call__(self): return [1,2,3] @@ -212,7 +212,7 @@ raise AssertionError("Did not raise") def test_remote_classmethod(self): - class A: + class A(object): z = 8 @classmethod @@ -226,7 +226,7 @@ assert res == 8 def test_types_reverse_mapping(self): - class A: + class A(object): def m(self, tp): assert type(self) is tp @@ -237,7 +237,7 @@ xa.m(xA) def test_instantiate_remote_type(self): - class C: + class C(object): def __init__(self, y): self.y = y From arigo at codespeak.net Wed Jul 16 23:43:49 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 16 Jul 2008 23:43:49 +0200 (CEST) Subject: [pypy-svn] r56605 - in pypy/dist/pypy: lang/smalltalk lang/smalltalk/test lang/smalltalk/tool lib lib/xml lib/xml/dom lib/xml/etree lib/xml/parsers lib/xml/sax rlib/rsdl translator/goal translator/oosupport/test_template Message-ID: <20080716214349.83DA369804D@codespeak.net> Author: arigo Date: Wed Jul 16 23:43:48 2008 New Revision: 56605 Modified: pypy/dist/pypy/lang/smalltalk/test/test_wrapper.py (props changed) pypy/dist/pypy/lang/smalltalk/todo.txt (props changed) pypy/dist/pypy/lang/smalltalk/tool/infostats.py (props changed) pypy/dist/pypy/lang/smalltalk/wrapper.py (props changed) pypy/dist/pypy/lib/_hashlib.py (props changed) pypy/dist/pypy/lib/dbm.py (contents, props changed) pypy/dist/pypy/lib/functools.py (props changed) pypy/dist/pypy/lib/xml/ (props changed) pypy/dist/pypy/lib/xml/dom/ (props changed) pypy/dist/pypy/lib/xml/etree/ (props changed) pypy/dist/pypy/lib/xml/parsers/ (props changed) pypy/dist/pypy/lib/xml/sax/ (props changed) pypy/dist/pypy/rlib/rsdl/eci.py (props changed) pypy/dist/pypy/translator/goal/targetgbimplementation.py (props changed) pypy/dist/pypy/translator/goal/targetgbrom4.py (props changed) pypy/dist/pypy/translator/goal/targetimageloadingsmalltalk.py (props changed) pypy/dist/pypy/translator/goal/targetsimpleread.py (props changed) pypy/dist/pypy/translator/goal/targetsimplevideo.py (props changed) pypy/dist/pypy/translator/goal/targetsimplewrite.py (props changed) pypy/dist/pypy/translator/oosupport/test_template/exception.py (props changed) Log: fixeol Modified: pypy/dist/pypy/lib/dbm.py ============================================================================== --- pypy/dist/pypy/lib/dbm.py (original) +++ pypy/dist/pypy/lib/dbm.py Wed Jul 16 23:43:48 2008 @@ -1,195 +1,195 @@ -from ctypes import * -import ctypes.util -import os - -_singleton = 'one' - -class error(Exception): - def __init__(self, msg): - self.msg = msg - - def __str__(self): - return self.msg - -class datum(Structure): - _fields_ = [ - ('dptr', c_char_p), - ('dsize', c_int), - ] - -class dbm(object): - def __init__(self, dbmobj): - self._aobj = dbmobj - - def close(self): - getattr(lib, funcs['close'])(self._aobj) - self._aobj = None - - def keys(self): - if not self._aobj: - raise error('DBM object has already been closed') - allkeys = [] - k = getattr(lib, funcs['firstkey'])(self._aobj) - while k.dptr: - allkeys.append(k.dptr[:k.dsize]) - k = getattr(lib, funcs['nextkey'])(self._aobj) - return allkeys - - def get(self, key, default=_singleton): - if not self._aobj: - raise error('DBM object has already been closed') - dat = datum() - dat.dptr = c_char_p(key) - dat.dsize = c_int(len(key)) - k = getattr(lib, funcs['fetch'])(self._aobj, dat) - if k.dptr: - return k.dptr[:k.dsize] - if default is _singleton: - raise KeyError - if getattr(lib, funcs['error'])(self._aobj): - getattr(lib, funcs['clearerr'])(self._aobj) - raise error("") - return default - - def __len__(self): - return len(self.keys()) - - def __getitem__(self, key): - assert isinstance(key, str) - value = self.get(key) - if value is None: - raise KeyError - - def _set(self, key, value): - if not self._aobj: - raise error('DBM object has already been closed') - if not isinstance(key, str): - raise TypeError("dbm mappings have string indices only") - dat = datum() - dat.dptr = c_char_p(key) - dat.dsize = c_int(len(key)) - if value == None: - status = getattr(lib, funcs['delete'])(self._aobj, dat) - if status < 0: - getattr(lib, funcs['clearerr'])(self._aobj) - raise KeyError(key) - else: - if not isinstance(value, str): - raise TypeError("dbm mappings have string indices only") - data = datum() - data.dptr = c_char_p(value) - data.dsize = c_int(len(value)) - status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_INSERT) - if status == 1: - status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_REPLACE) - if getattr(lib, funcs['error'])(self._aobj): - getattr(lib, funcs['clearerr'])(self._aobj) - raise error("") - return status - - def setdefault(self, key, default=None): - if not self._aobj: - raise error('DBM object has already been closed') - dat = datum() - dat.dptr = c_char_p(key) - dat.dsize = c_int(len(key)) - k = getattr(lib, funcs['fetch'])(self._aobj, dat) - if k.dptr: - return k.dptr[:k.dsize] - if default: - data = datum() - data.dptr = c_char_p(default) - data.dsize = c_int(len(default)) - status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_INSERT) - if status < 0: - getattr(lib, funcs['clearerr'])(self._aobj) - raise error("cannot add item to database") - return default - return None - - def has_key(self, key): - if not self._aobj: - raise error('DBM object has already been closed') - dat = datum() - dat.dptr = c_char_p(key) - dat.dsize = c_int(len(key)) - k = getattr(lib, funcs['fetch'])(self._aobj, dat) - if k.dptr: - return True - return False - - def __setitem__(self, key, value): - if not isinstance(key, str) and isinstance(value, str): - raise error("dbm mappings have string indices only") - self._set(key, value) - -# initialization for Berkeley DB -_bdb_funcs = { - 'open': '__db_ndbm_open', - 'close': '__db_ndbm_close', - 'firstkey': '__db_ndbm_firstkey', - 'nextkey': '__db_ndbm_nextkey', - 'fetch': '__db_ndbm_fetch', - 'store': '__db_ndbm_store', - 'error': '__db_ndbm_error', - 'delete': '__db_ndbm_delete', -} - -_normal_funcs = { - 'open': 'dbm_open', - 'close': 'dbm_close', - 'firstkey': 'dbm_firstkey', - 'nextkey': 'dbm_nextkey', - 'fetch': 'dbm_fetch', - 'store': 'dbm_store', - 'error': 'dbm_error', - 'delete': 'dbm_delete', -} - -try: - libpath = ctypes.util.find_library('db') - if not libpath: raise - lib = CDLL(libpath) # Linux - _platform = 'bdb' - lib.__db_ndbm_open.argtypes = [c_char_p, c_int, c_int] - lib.__db_ndbm_close.restype = c_void_p - lib.__db_ndbm_firstkey.restype = datum - lib.__db_ndbm_nextkey.restype = datum - lib.__db_ndbm_fetch.restype = datum - lib.__db_ndbm_store.restype = c_int - funcs = _bdb_funcs -except: - lib = CDLL("/usr/lib/libdbm.dylib") # OS X - _platform = 'osx' - lib.dbm_open.argtypes = [c_char_p, c_int, c_int] - lib.dbm_close.restype = c_void_p - lib.dbm_firstkey.restype = datum - lib.dbm_nextkey.restype = datum - lib.dbm_fetch.restype = datum - lib.dbm_store.restype = c_int - funcs = _normal_funcs - - - -lib.DBM_INSERT = 0 -lib.DBM_REPLACE = 1 - -def open(filename, flag='r', mode=0666): - "open a DBM database" - openflag = 0 - - try: - openflag = { - 'r': os.O_RDONLY, - 'rw': os.O_RDWR, - 'w': os.O_RDWR | os.O_CREAT, - 'c': os.O_RDWR | os.O_CREAT, - 'n': os.O_RDWR | os.O_CREAT | os.O_TRUNC, - }[flag] - except KeyError, e: - raise error("arg 2 to open should be 'r', 'w', 'c', or 'n'") - - a_db = getattr(lib, funcs['open'])(filename, openflag, mode) - if a_db == 0: - raise error("Could not open file %s.db" % filename) - return dbm(a_db) +from ctypes import * +import ctypes.util +import os + +_singleton = 'one' + +class error(Exception): + def __init__(self, msg): + self.msg = msg + + def __str__(self): + return self.msg + +class datum(Structure): + _fields_ = [ + ('dptr', c_char_p), + ('dsize', c_int), + ] + +class dbm(object): + def __init__(self, dbmobj): + self._aobj = dbmobj + + def close(self): + getattr(lib, funcs['close'])(self._aobj) + self._aobj = None + + def keys(self): + if not self._aobj: + raise error('DBM object has already been closed') + allkeys = [] + k = getattr(lib, funcs['firstkey'])(self._aobj) + while k.dptr: + allkeys.append(k.dptr[:k.dsize]) + k = getattr(lib, funcs['nextkey'])(self._aobj) + return allkeys + + def get(self, key, default=_singleton): + if not self._aobj: + raise error('DBM object has already been closed') + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + k = getattr(lib, funcs['fetch'])(self._aobj, dat) + if k.dptr: + return k.dptr[:k.dsize] + if default is _singleton: + raise KeyError + if getattr(lib, funcs['error'])(self._aobj): + getattr(lib, funcs['clearerr'])(self._aobj) + raise error("") + return default + + def __len__(self): + return len(self.keys()) + + def __getitem__(self, key): + assert isinstance(key, str) + value = self.get(key) + if value is None: + raise KeyError + + def _set(self, key, value): + if not self._aobj: + raise error('DBM object has already been closed') + if not isinstance(key, str): + raise TypeError("dbm mappings have string indices only") + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + if value == None: + status = getattr(lib, funcs['delete'])(self._aobj, dat) + if status < 0: + getattr(lib, funcs['clearerr'])(self._aobj) + raise KeyError(key) + else: + if not isinstance(value, str): + raise TypeError("dbm mappings have string indices only") + data = datum() + data.dptr = c_char_p(value) + data.dsize = c_int(len(value)) + status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_INSERT) + if status == 1: + status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_REPLACE) + if getattr(lib, funcs['error'])(self._aobj): + getattr(lib, funcs['clearerr'])(self._aobj) + raise error("") + return status + + def setdefault(self, key, default=None): + if not self._aobj: + raise error('DBM object has already been closed') + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + k = getattr(lib, funcs['fetch'])(self._aobj, dat) + if k.dptr: + return k.dptr[:k.dsize] + if default: + data = datum() + data.dptr = c_char_p(default) + data.dsize = c_int(len(default)) + status = getattr(lib, funcs['store'])(self._aobj, dat, data, lib.DBM_INSERT) + if status < 0: + getattr(lib, funcs['clearerr'])(self._aobj) + raise error("cannot add item to database") + return default + return None + + def has_key(self, key): + if not self._aobj: + raise error('DBM object has already been closed') + dat = datum() + dat.dptr = c_char_p(key) + dat.dsize = c_int(len(key)) + k = getattr(lib, funcs['fetch'])(self._aobj, dat) + if k.dptr: + return True + return False + + def __setitem__(self, key, value): + if not isinstance(key, str) and isinstance(value, str): + raise error("dbm mappings have string indices only") + self._set(key, value) + +# initialization for Berkeley DB +_bdb_funcs = { + 'open': '__db_ndbm_open', + 'close': '__db_ndbm_close', + 'firstkey': '__db_ndbm_firstkey', + 'nextkey': '__db_ndbm_nextkey', + 'fetch': '__db_ndbm_fetch', + 'store': '__db_ndbm_store', + 'error': '__db_ndbm_error', + 'delete': '__db_ndbm_delete', +} + +_normal_funcs = { + 'open': 'dbm_open', + 'close': 'dbm_close', + 'firstkey': 'dbm_firstkey', + 'nextkey': 'dbm_nextkey', + 'fetch': 'dbm_fetch', + 'store': 'dbm_store', + 'error': 'dbm_error', + 'delete': 'dbm_delete', +} + +try: + libpath = ctypes.util.find_library('db') + if not libpath: raise + lib = CDLL(libpath) # Linux + _platform = 'bdb' + lib.__db_ndbm_open.argtypes = [c_char_p, c_int, c_int] + lib.__db_ndbm_close.restype = c_void_p + lib.__db_ndbm_firstkey.restype = datum + lib.__db_ndbm_nextkey.restype = datum + lib.__db_ndbm_fetch.restype = datum + lib.__db_ndbm_store.restype = c_int + funcs = _bdb_funcs +except: + lib = CDLL("/usr/lib/libdbm.dylib") # OS X + _platform = 'osx' + lib.dbm_open.argtypes = [c_char_p, c_int, c_int] + lib.dbm_close.restype = c_void_p + lib.dbm_firstkey.restype = datum + lib.dbm_nextkey.restype = datum + lib.dbm_fetch.restype = datum + lib.dbm_store.restype = c_int + funcs = _normal_funcs + + + +lib.DBM_INSERT = 0 +lib.DBM_REPLACE = 1 + +def open(filename, flag='r', mode=0666): + "open a DBM database" + openflag = 0 + + try: + openflag = { + 'r': os.O_RDONLY, + 'rw': os.O_RDWR, + 'w': os.O_RDWR | os.O_CREAT, + 'c': os.O_RDWR | os.O_CREAT, + 'n': os.O_RDWR | os.O_CREAT | os.O_TRUNC, + }[flag] + except KeyError, e: + raise error("arg 2 to open should be 'r', 'w', 'c', or 'n'") + + a_db = getattr(lib, funcs['open'])(filename, openflag, mode) + if a_db == 0: + raise error("Could not open file %s.db" % filename) + return dbm(a_db) From bgola at codespeak.net Thu Jul 17 00:09:10 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Thu, 17 Jul 2008 00:09:10 +0200 (CEST) Subject: [pypy-svn] r56607 - in pypy/branch/2.5-features/pypy/interpreter: astcompiler pyparser test Message-ID: <20080716220910.F02872A80B0@codespeak.net> Author: bgola Date: Thu Jul 17 00:09:08 2008 New Revision: 56607 Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py pypy/branch/2.5-features/pypy/interpreter/test/test_compiler.py Log: fixing some tests under interpreter/test/ Modified: pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/astcompiler/pycodegen.py Thu Jul 17 00:09:08 2008 @@ -333,18 +333,21 @@ self.set_lineno(node) for default in node.defaults: default.accept( self ) + self._makeClosure(gen, len(node.defaults)) + for i in range(ndecorators): + self.emitop_int('CALL_FUNCTION', 1) + + def _makeClosure(self, gen, args): frees = gen.scope.get_free_vars_in_parent() if frees: for name in frees: self.emitop('LOAD_CLOSURE', name) + self.emitop_int('BUILD_TUPLE', len(frees)) self.emitop_code('LOAD_CONST', gen) - self.emitop_int('MAKE_CLOSURE', len(node.defaults)) + self.emitop_int('MAKE_CLOSURE', args) else: self.emitop_code('LOAD_CONST', gen) - self.emitop_int('MAKE_FUNCTION', len(node.defaults)) - - for i in range(ndecorators): - self.emitop_int('CALL_FUNCTION', 1) + self.emitop_int('MAKE_FUNCTION', args) def visitClass(self, node): gen = ClassCodeGenerator(self.space, node, @@ -356,15 +359,7 @@ for base in node.bases: base.accept( self ) self.emitop_int('BUILD_TUPLE', len(node.bases)) - frees = gen.scope.get_free_vars_in_parent() - if frees: - for name in frees: - self.emitop('LOAD_CLOSURE', name) - self.emitop_code('LOAD_CONST', gen) - self.emitop_int('MAKE_CLOSURE', 0) - else: - self.emitop_code('LOAD_CONST', gen) - self.emitop_int('MAKE_FUNCTION', 0) + self._makeClosure(gen, 0) self.emitop_int('CALL_FUNCTION', 0) self.emit('BUILD_CLASS') self.storeName(node.name, node.lineno) @@ -658,16 +653,7 @@ inner.accept( gen ) gen.finish() self.set_lineno(node) - frees = gen.scope.get_free_vars_in_parent() - if frees: - for name in frees: - self.emitop('LOAD_CLOSURE', name) - self.emitop_code('LOAD_CONST', gen) - self.emitop_int('MAKE_CLOSURE', 0) - else: - self.emitop_code('LOAD_CONST', gen) - self.emitop_int('MAKE_FUNCTION', 0) - + self._makeClosure(gen, 0) # precomputation of outmost iterable qual0 = inner.quals[0] assert isinstance(qual0, ast.GenExprFor) @@ -1167,10 +1153,11 @@ self.emit('RETURN_VALUE') def visitYield(self, node): - if len(self.setups): - kind, block = self.setups[-1] - if kind == TRY_FINALLY: - raise SyntaxError("'yield' not allowed in a 'try' block " + if self.space.config.objspace.pyversion < '2.5': + if len(self.setups): + kind, block = self.setups[-1] + if kind == TRY_FINALLY: + raise SyntaxError("'yield' not allowed in a 'try' block " "with a 'finally' clause", node.lineno) self.set_lineno(node) Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyparser/astbuilder.py Thu Jul 17 00:09:08 2008 @@ -837,7 +837,7 @@ """ import_from: 'from' dotted_name 'import' ('*' | '(' import_as_names [','] ')' | import_as_names) - import_as_names: import_as_name (',' import_as_name)* + import_as_names: import_as_name (',' import_as_name)* [','] import_as_name: NAME [NAME NAME] """ atoms = get_atoms(builder, nb) @@ -855,6 +855,10 @@ tokens = slicecut( atoms, index+1, -1 ) else: tokens = atoms[index:] + if tokens[-1].name == builder.parser.tokens['COMMA']: + raise SyntaxError, "trailing comma not allowed without" \ + "surrounding parentheses" + index = 0 l = len(tokens) names = [] Modified: pypy/branch/2.5-features/pypy/interpreter/test/test_compiler.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/test/test_compiler.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/test/test_compiler.py Thu Jul 17 00:09:08 2008 @@ -582,10 +582,6 @@ assert space.float_w(w_result) == 0 -class TestECCompiler(BaseTestCompiler): - def setup_method(self, method): - self.compiler = self.space.getexecutioncontext().compiler - class TestPyCCompiler(BaseTestCompiler): def setup_method(self, method): self.compiler = CPythonCompiler(self.space) @@ -600,16 +596,15 @@ def skip_on_2_4(self): py.test.skip("syntax not supported by the CPython 2.4 compiler") test_continue_in_nested_finally = skip_on_2_4 + elif sys.version_info > (2, 4): + def skip_on_2_5(self): + py.test.skip("syntax changed in CPython 2.5 compiler") + test_yield_in_finally = skip_on_2_5 + - -class TestPythonAstCompiler(BaseTestCompiler): +class TestPythonAstCompiler_25_grammar(BaseTestCompiler): def setup_method(self, method): - self.compiler = PythonAstCompiler(self.space, "2.4") - - -class TestPythonAstCompiler_25_grammar: - def setup_method(self, method): - self.compiler = PythonAstCompiler(self.space, "2.5a") + self.compiler = PythonAstCompiler(self.space, "2.5") def test_from_future_import(self): source = """from __future__ import with_statement @@ -626,6 +621,22 @@ assert isinstance(code, PyCode) assert code.co_filename == '' + def test_yield_in_finally(self): # behavior changed in 2.5 + code ='def f():\n try:\n yield 19\n finally:\n pass\n' + self.compiler.compile(code, '', 'single', 0) + + +class TestECCompiler(BaseTestCompiler): + def setup_method(self, method): + self.space.config.objspace.pyversion = "2.4" + self.compiler = self.space.getexecutioncontext().compiler + + +class TestPythonAstCompiler(BaseTestCompiler): + def setup_method(self, method): + self.space.config.objspace.pyversion = "2.4" + self.compiler = PythonAstCompiler(self.space, "2.4") + class AppTestOptimizer: def test_constant_fold_add(self): From arigo at codespeak.net Thu Jul 17 10:58:14 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 17 Jul 2008 10:58:14 +0200 (CEST) Subject: [pypy-svn] r56609 - pypy/dist/pypy/lib/distributed/test Message-ID: <20080717085814.35747169F05@codespeak.net> Author: arigo Date: Thu Jul 17 10:58:13 2008 New Revision: 56609 Modified: pypy/dist/pypy/lib/distributed/test/test_socklayer.py Log: Grr. One more missed place where we need a new-style class. Modified: pypy/dist/pypy/lib/distributed/test/test_socklayer.py ============================================================================== --- pypy/dist/pypy/lib/distributed/test/test_socklayer.py (original) +++ pypy/dist/pypy/lib/distributed/test/test_socklayer.py Thu Jul 17 10:58:13 2008 @@ -9,7 +9,7 @@ "usemodules":("_stackless","_socket", "select")}) def test_socklayer(self): - class X: + class X(object): z = 3 x = X() From hpk at codespeak.net Thu Jul 17 11:13:46 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Thu, 17 Jul 2008 11:13:46 +0200 (CEST) Subject: [pypy-svn] r56610 - pypy/dist/pypy/doc Message-ID: <20080717091346.7EC53498073@codespeak.net> Author: hpk Date: Thu Jul 17 11:13:42 2008 New Revision: 56610 Modified: pypy/dist/pypy/doc/faq.txt Log: * trying to make RPython section slightly more readable for people not knowing anything about pypy * add an entry regarding Zope's restricted python Modified: pypy/dist/pypy/doc/faq.txt ============================================================================== --- pypy/dist/pypy/doc/faq.txt (original) +++ pypy/dist/pypy/doc/faq.txt Thu Jul 17 11:13:42 2008 @@ -249,17 +249,24 @@ Report from the PyPy project to the E.U., http://codespeak.net/svn/pypy/extradoc/eu-report/D05.1_Publish_on_translating_a_very-high-level_description.pdf +.. _`PyPy's RPython`: + ------------------------------ What is this RPython language? ------------------------------ -RPython is a restricted subset of the Python language. The restrictions are to -ensure that type inference (and so, ultimately, translation to other languages) of -the program is possible. These restrictions only apply after the full import -happens, so at import time arbitrary Python code can be executed. Another -important point is that the property of "being RPython" always applies to a full -program, not to single functions or modules (the translation tool chain does a -full program analysis). +RPython is a restricted subset of the Python language. It is used for +implementing dynamic language interpreters within the PyPy framework. The +restrictions are to ensure that type inference (and so, ultimately, translation +to other languages) of RPython programs is possible. These restrictions only +apply after the full import happens, so at import time arbitrary Python code can +be executed. + +The property of "being RPython" always applies to a full program, not to single +functions or modules (the translation tool chain does a full program analysis). +"Full program" in the context of "being RPython" is all the code reachable from +an "entry point" function. The translation toolchain follows all calls +recursively and discovers what belongs to the program and what not. The restrictions that apply to programs to be RPython mostly limit the ability of mixing types in arbitrary ways. RPython does not allow the usage of two @@ -277,14 +284,17 @@ .. _`RPython description`: coding-guide.html#restricted-python +------------------------------------------------------------------------- +Does RPython have anything to do with Zope's Restricted Python? +------------------------------------------------------------------------- --------------------------------------------------------------------------- -What do you mean by "full program"? All the code in all the modules I use? --------------------------------------------------------------------------- +No. `Zope's RestrictedPython`_ aims to provide a sandboxed +execution environment for CPython. `PyPy's RPython`_ is the implementation +language for dynamic language interpreters. However, PyPy also provides +a robust `sandboxed Python Interpreter`_. -"Full program" in the context of "being RPython" is all the code reachable from -an "entry point" function. The translation toolchain follows all calls -recursively and discovers what belongs to the program and what not. +.. _`sandboxed Python Interpreter`: sandbox.html +.. _`Zope's RestrictedPython`: http://pypi.python.org/pypi/RestrictedPython ------------------------------------------------------------------------- Can I use PyPy and RPython to compile smaller parts of my Python program? From arigo at codespeak.net Thu Jul 17 11:20:25 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 17 Jul 2008 11:20:25 +0200 (CEST) Subject: [pypy-svn] r56611 - pypy/dist/pypy/objspace/std Message-ID: <20080717092025.2CACE498073@codespeak.net> Author: arigo Date: Thu Jul 17 11:20:24 2008 New Revision: 56611 Modified: pypy/dist/pypy/objspace/std/unicodeobject.py pypy/dist/pypy/objspace/std/unicodetype.py Log: Fix test_ropeunicodeobject by removing some code duplication and using an explicit fall-back from unicode.decode() to str.decode(). This is a slightly obscure method so a general fallback that is a bit slower but works for any unicode implementation is good enough IMHO. Modified: pypy/dist/pypy/objspace/std/unicodeobject.py ============================================================================== --- pypy/dist/pypy/objspace/std/unicodeobject.py (original) +++ pypy/dist/pypy/objspace/std/unicodeobject.py Thu Jul 17 11:20:24 2008 @@ -773,14 +773,6 @@ w_retval = encode_object(space, w_unistr, encoding, errors) return w_retval -def unicode_decode__Unicode_ANY_ANY(space, w_unicode, w_encoding=None, w_errors=None): - from pypy.objspace.std.unicodetype import _get_encoding_and_errors,\ - decode_object, unicode_from_string - encoding, errors = _get_encoding_and_errors(space, w_encoding, w_errors) - if encoding is None and errors is None: - return unicode_from_string(space, w_unicode) - return decode_object(space, w_unicode, encoding, errors) - def unicode_partition__Unicode_Unicode(space, w_unistr, w_unisub): unistr = w_unistr._value unisub = w_unisub._value Modified: pypy/dist/pypy/objspace/std/unicodetype.py ============================================================================== --- pypy/dist/pypy/objspace/std/unicodetype.py (original) +++ pypy/dist/pypy/objspace/std/unicodetype.py Thu Jul 17 11:20:24 2008 @@ -1,5 +1,6 @@ from pypy.interpreter import gateway from pypy.objspace.std.stdtypedef import * +from pypy.objspace.std.register_all import register_all from pypy.objspace.std.basestringtype import basestring_typedef from pypy.interpreter.error import OperationError @@ -245,8 +246,13 @@ # raising UnicodeDecodeError is messy, "please crash for me" return unicode_from_encoded_object(space, w_str, "ascii", "strict") +def unicode_decode__unitypedef_ANY_ANY(space, w_unicode, w_encoding=None, + w_errors=None): + return space.call_method(space.str(w_unicode), 'decode', + w_encoding, w_errors) -def descr__new__(space, w_unicodetype, w_string='', w_encoding=None, w_errors=None): + +def descr_new_(space, w_unicodetype, w_string='', w_encoding=None, w_errors=None): # NB. the default value of w_obj is really a *wrapped* empty string: # there is gateway magic at work from pypy.objspace.std.unicodeobject import W_UnicodeObject @@ -285,7 +291,7 @@ # ____________________________________________________________ unicode_typedef = StdTypeDef("unicode", basestring_typedef, - __new__ = newmethod(descr__new__), + __new__ = newmethod(descr_new_), __doc__ = '''unicode(string [, encoding[, errors]]) -> object Create a new Unicode object from the given encoded string. @@ -295,3 +301,6 @@ unicode_typedef.custom_hash = True unicode_typedef.registermethods(globals()) + +unitypedef = unicode_typedef +register_all(vars(), globals()) From arigo at codespeak.net Thu Jul 17 11:23:54 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 17 Jul 2008 11:23:54 +0200 (CEST) Subject: [pypy-svn] r56612 - pypy/branch/opt-option Message-ID: <20080717092354.912BC498073@codespeak.net> Author: arigo Date: Thu Jul 17 11:23:53 2008 New Revision: 56612 Added: pypy/branch/opt-option/ - copied from r56611, pypy/dist/ Log: A branch in which to try to implement the --opt translation option. From arigo at codespeak.net Thu Jul 17 11:53:25 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 17 Jul 2008 11:53:25 +0200 (CEST) Subject: [pypy-svn] r56613 - pypy/dist/pypy/config Message-ID: <20080717095325.4243F2A00E2@codespeak.net> Author: arigo Date: Thu Jul 17 11:53:24 2008 New Revision: 56613 Modified: pypy/dist/pypy/config/pypyoption.py Log: This is no longer broken as far as I can tell. Modified: pypy/dist/pypy/config/pypyoption.py ============================================================================== --- pypy/dist/pypy/config/pypyoption.py (original) +++ pypy/dist/pypy/config/pypyoption.py Thu Jul 17 11:53:24 2008 @@ -285,7 +285,7 @@ "special case the 'list[integer]' expressions", default=False), BoolOption("builtinshortcut", - "a shortcut for operations between built-in types XXX BROKEN", + "a shortcut for operations between built-in types", default=False), BoolOption("getattributeshortcut", "track types that override __getattribute__", From hpk at codespeak.net Thu Jul 17 13:59:20 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Thu, 17 Jul 2008 13:59:20 +0200 (CEST) Subject: [pypy-svn] r56615 - in pypy/dist/pypy: config translator/tool Message-ID: <20080717115920.C4C642A018B@codespeak.net> Author: hpk Date: Thu Jul 17 13:59:18 2008 New Revision: 56615 Modified: pypy/dist/pypy/config/pypyoption.py pypy/dist/pypy/translator/tool/cbuild.py Log: whack a bit to make an --allworkingmodules built work on solaris 5.10 Modified: pypy/dist/pypy/config/pypyoption.py ============================================================================== --- pypy/dist/pypy/config/pypyoption.py (original) +++ pypy/dist/pypy/config/pypyoption.py Thu Jul 17 13:59:18 2008 @@ -37,6 +37,15 @@ del working_modules["termios"] del working_modules["_minimal_curses"] +if sys.platform == "sunos5": + del working_modules['mmap'] # depend on ctypes, can'T get at c-level 'errono' + del working_modules['rctime'] # depend on ctypes, missing tm_zone/tm_gmtoff + del working_modules['signal'] # depend on ctypes, can'T get at c-level 'errono' + del working_modules['fcntl'] # LOCK_NB not defined + del working_modules["_minimal_curses"] + del working_modules["termios"] + + module_dependencies = {} module_suggests = { # the reason you want _rawffi is for ctypes, which Modified: pypy/dist/pypy/translator/tool/cbuild.py ============================================================================== --- pypy/dist/pypy/translator/tool/cbuild.py (original) +++ pypy/dist/pypy/translator/tool/cbuild.py Thu Jul 17 13:59:18 2008 @@ -505,8 +505,16 @@ self.libraries.append('m') if 'pthread' not in self.libraries: self.libraries.append('pthread') - self.compile_extra += CFLAGS + ['-fomit-frame-pointer', '-pthread'] - self.link_extra += ['-pthread'] + + self.compile_extra += CFLAGS + ['-fomit-frame-pointer'] + if sys.platform != 'sunos5': + self.compile_extra += ['-pthread'] + self.link_extra += ['-pthread'] + else: + self.compile_extra += ['-pthreads'] + # XXX probably socket and nsl should rather go + # to be defined in the socket module + self.link_extra += ['-lpthread', '-lsocket', '-lnsl'] if sys.platform == 'win32': self.link_extra += ['/DEBUG'] # generate .pdb file if sys.platform == 'darwin': From hpk at codespeak.net Fri Jul 18 11:59:20 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Fri, 18 Jul 2008 11:59:20 +0200 (CEST) Subject: [pypy-svn] r56637 - in pypy/dist/pypy: rlib translator/tool Message-ID: <20080718095920.C6C6C169FCF@codespeak.net> Author: hpk Date: Fri Jul 18 11:59:18 2008 New Revision: 56637 Modified: pypy/dist/pypy/rlib/_rsocket_rffi.py pypy/dist/pypy/translator/tool/cbuild.py Log: move defs for sunos5 libs needed for sockets to a better place (thanks, exarkun) Modified: pypy/dist/pypy/rlib/_rsocket_rffi.py ============================================================================== --- pypy/dist/pypy/rlib/_rsocket_rffi.py (original) +++ pypy/dist/pypy/rlib/_rsocket_rffi.py Fri Jul 18 11:59:18 2008 @@ -7,11 +7,11 @@ from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.rlib.rarithmetic import intmask, r_uint -import os +import os,sys _POSIX = os.name == "posix" _MS_WINDOWS = os.name == "nt" - +_SOLARIS = sys.platform == "sunos5" if _POSIX: includes = ('sys/types.h', @@ -36,6 +36,10 @@ HEADER = ''.join(['#include <%s>\n' % filename for filename in includes]) COND_HEADER = ''.join(['#ifdef %s\n#include <%s>\n#endif\n' % cond_include for cond_include in cond_includes]) + +if _SOLARIS: + libraries = libraries + ('socket', 'nsl') + if _MS_WINDOWS: includes = () libraries = ('ws2_32',) Modified: pypy/dist/pypy/translator/tool/cbuild.py ============================================================================== --- pypy/dist/pypy/translator/tool/cbuild.py (original) +++ pypy/dist/pypy/translator/tool/cbuild.py Fri Jul 18 11:59:18 2008 @@ -503,18 +503,15 @@ if not sys.platform in ('win32', 'darwin'): # xxx if 'm' not in self.libraries: self.libraries.append('m') + self.compile_extra += CFLAGS + ['-fomit-frame-pointer'] if 'pthread' not in self.libraries: self.libraries.append('pthread') - - self.compile_extra += CFLAGS + ['-fomit-frame-pointer'] if sys.platform != 'sunos5': self.compile_extra += ['-pthread'] self.link_extra += ['-pthread'] else: self.compile_extra += ['-pthreads'] - # XXX probably socket and nsl should rather go - # to be defined in the socket module - self.link_extra += ['-lpthread', '-lsocket', '-lnsl'] + self.link_extra += ['-lpthread'] if sys.platform == 'win32': self.link_extra += ['/DEBUG'] # generate .pdb file if sys.platform == 'darwin': From arigo at codespeak.net Fri Jul 18 23:01:02 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 18 Jul 2008 23:01:02 +0200 (CEST) Subject: [pypy-svn] r56647 - pypy/dist/pypy/rlib/test Message-ID: <20080718210102.51702169EA7@codespeak.net> Author: arigo Date: Fri Jul 18 23:00:58 2008 New Revision: 56647 Modified: pypy/dist/pypy/rlib/test/test_libffi.py Log: Port this test to 64 bits. Modified: pypy/dist/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/dist/pypy/rlib/test/test_libffi.py (original) +++ pypy/dist/pypy/rlib/test/test_libffi.py Fri Jul 18 23:00:58 2008 @@ -162,25 +162,26 @@ a2 = rffi.cast(rffi.INTP, p_a2)[0] res = rffi.cast(rffi.INTP, ll_res) if a1 > a2: - res[0] = 1 + res[0] = rffi.cast(rffi.INT, 1) else: - res[0] = -1 + res[0] = rffi.cast(rffi.INT, -1) ptr = CallbackFuncPtr([ffi_type_pointer, ffi_type_pointer], ffi_type_sint, callback) TP = rffi.CArray(rffi.INT) to_sort = lltype.malloc(TP, 4, flavor='raw') - to_sort[0] = 4 - to_sort[1] = 3 - to_sort[2] = 1 - to_sort[3] = 2 + to_sort[0] = rffi.cast(rffi.INT, 4) + to_sort[1] = rffi.cast(rffi.INT, 3) + to_sort[2] = rffi.cast(rffi.INT, 1) + to_sort[3] = rffi.cast(rffi.INT, 2) qsort.push_arg(rffi.cast(rffi.VOIDP, to_sort)) qsort.push_arg(rffi.sizeof(rffi.INT)) qsort.push_arg(4) qsort.push_arg(ptr.ll_closure) qsort.call(lltype.Void) - assert [to_sort[i] for i in range(4)] == [1,2,3,4] + assert ([rffi.cast(lltype.Signed, to_sort[i]) for i in range(4)] == + [1,2,3,4]) lltype.free(to_sort, flavor='raw') keepalive_until_here(ptr) # <= this test is not translated, but don't # forget this in code that is meant to be From arigo at codespeak.net Fri Jul 18 23:16:17 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 18 Jul 2008 23:16:17 +0200 (CEST) Subject: [pypy-svn] r56650 - in pypy/dist/pypy/module/_rawffi: . test Message-ID: <20080718211617.78C90169F18@codespeak.net> Author: arigo Date: Fri Jul 18 23:16:17 2008 New Revision: 56650 Modified: pypy/dist/pypy/module/_rawffi/callback.py pypy/dist/pypy/module/_rawffi/test/test__rawffi.py Log: Fix test_callback for 64-bit-ness. Fix the code to not truncate addresses to UINT (!). Modified: pypy/dist/pypy/module/_rawffi/callback.py ============================================================================== --- pypy/dist/pypy/module/_rawffi/callback.py (original) +++ pypy/dist/pypy/module/_rawffi/callback.py Fri Jul 18 23:16:17 2008 @@ -30,7 +30,7 @@ argtypes = callback_ptr.args space = callback_ptr.space try: - w_args = space.newlist([space.wrap(rffi.cast(rffi.UINT, ll_args[i])) + w_args = space.newlist([space.wrap(rffi.cast(rffi.ULONG, ll_args[i])) for i in range(len(argtypes))]) w_res = space.call(w_callable, w_args) if callback_ptr.result != 'O': # don't return void Modified: pypy/dist/pypy/module/_rawffi/test/test__rawffi.py ============================================================================== --- pypy/dist/pypy/module/_rawffi/test/test__rawffi.py (original) +++ pypy/dist/pypy/module/_rawffi/test/test__rawffi.py Fri Jul 18 23:16:17 2008 @@ -458,12 +458,11 @@ ll_to_sort = _rawffi.Array('i')(4) for i in range(4): ll_to_sort[i] = 4-i - qsort = libc.ptr('qsort', ['P', 'i', 'i', 'P'], None) - resarray = _rawffi.Array('i')(1) + qsort = libc.ptr('qsort', ['P', 'l', 'l', 'P'], None) bogus_args = [] def compare(a, b): - a1 = _rawffi.Array('i').fromaddress(_rawffi.Array('i').fromaddress(a, 1)[0], 1) - a2 = _rawffi.Array('i').fromaddress(_rawffi.Array('i').fromaddress(b, 1)[0], 1) + a1 = _rawffi.Array('i').fromaddress(_rawffi.Array('P').fromaddress(a, 1)[0], 1) + a2 = _rawffi.Array('i').fromaddress(_rawffi.Array('P').fromaddress(b, 1)[0], 1) print "comparing", a1[0], "with", a2[0] if a1[0] not in [1,2,3,4] or a2[0] not in [1,2,3,4]: bogus_args.append((a1[0], a2[0])) @@ -471,9 +470,9 @@ return 1 return -1 a1 = ll_to_sort.byptr() - a2 = _rawffi.Array('i')(1) + a2 = _rawffi.Array('l')(1) a2[0] = len(ll_to_sort) - a3 = _rawffi.Array('i')(1) + a3 = _rawffi.Array('l')(1) a3[0] = struct.calcsize('i') cb = _rawffi.CallbackPtr(compare, ['P', 'P'], 'i') a4 = cb.byptr() From arigo at codespeak.net Fri Jul 18 23:18:50 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 18 Jul 2008 23:18:50 +0200 (CEST) Subject: [pypy-svn] r56651 - pypy/dist/pypy/module/_rawffi/test Message-ID: <20080718211850.6E63C169F35@codespeak.net> Author: arigo Date: Fri Jul 18 23:18:49 2008 New Revision: 56651 Modified: pypy/dist/pypy/module/_rawffi/test/test__rawffi.py Log: Fix test for 64-bit-ness. Modified: pypy/dist/pypy/module/_rawffi/test/test__rawffi.py ============================================================================== --- pypy/dist/pypy/module/_rawffi/test/test__rawffi.py (original) +++ pypy/dist/pypy/module/_rawffi/test/test__rawffi.py Fri Jul 18 23:18:49 2008 @@ -592,16 +592,18 @@ def test_repr(self): import _rawffi, struct - s = struct.calcsize("i") + isize = struct.calcsize("i") + lsize = struct.calcsize("l") assert (repr(_rawffi.Array('i')) == - "<_rawffi.Array 'i' (%d, %d)>" % (s, s)) + "<_rawffi.Array 'i' (%d, %d)>" % (isize, isize)) # fragile S = _rawffi.Structure([('x', 'c'), ('y', 'l')]) - assert repr(_rawffi.Array((S, 2))) == "<_rawffi.Array 'V' (16, 4)>" + assert (repr(_rawffi.Array((S, 2))) == + "<_rawffi.Array 'V' (%d, %d)>" % (4*lsize, lsize)) assert (repr(_rawffi.Structure([('x', 'i'), ('yz', 'i')])) == - "<_rawffi.Structure 'x' 'yz' (%d, %d)>" % (2*s, s)) + "<_rawffi.Structure 'x' 'yz' (%d, %d)>" % (2*isize, isize)) s = _rawffi.Structure([('x', 'i'), ('yz', 'i')])() assert repr(s) == "<_rawffi struct %x>" % (s.buffer,) From antocuni at codespeak.net Sat Jul 19 11:11:44 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Sat, 19 Jul 2008 11:11:44 +0200 (CEST) Subject: [pypy-svn] r56659 - pypy/branch/oo-jit/pypy/jit/timeshifter Message-ID: <20080719091144.8C7DD16853F@codespeak.net> Author: antocuni Date: Sat Jul 19 11:11:41 2008 New Revision: 56659 Modified: pypy/branch/oo-jit/pypy/jit/timeshifter/rtimeshift.py Log: comment out this assert, as it makes all tests failing. Maciek, what was the idea beyond it? Modified: pypy/branch/oo-jit/pypy/jit/timeshifter/rtimeshift.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/timeshifter/rtimeshift.py (original) +++ pypy/branch/oo-jit/pypy/jit/timeshifter/rtimeshift.py Sat Jul 19 11:11:41 2008 @@ -738,7 +738,7 @@ class PromotionPathNode(AbstractPromotionPath): def __init__(self, next): - assert next + #assert next self.next = next def follow_path(self, path): path.append(self) From arigo at codespeak.net Sat Jul 19 11:33:05 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Sat, 19 Jul 2008 11:33:05 +0200 (CEST) Subject: [pypy-svn] r56660 - in pypy/dist/pypy/module/itertools: . test Message-ID: <20080719093305.9FAA9169F57@codespeak.net> Author: arigo Date: Sat Jul 19 11:33:04 2008 New Revision: 56660 Modified: pypy/dist/pypy/module/itertools/interp_itertools.py pypy/dist/pypy/module/itertools/test/test_itertools.py Log: Test and fix. Modified: pypy/dist/pypy/module/itertools/interp_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/interp_itertools.py (original) +++ pypy/dist/pypy/module/itertools/interp_itertools.py Sat Jul 19 11:33:04 2008 @@ -295,11 +295,11 @@ raise OperationError(space.w_TypeError, space.wrap("islice() takes at most 4 arguments (" + str(num_args) + " given)")) if space.is_w(w_stop, space.w_None): - stop = 0 - self.stoppable = False + stop = -1 + stoppable = False else: stop = space.int_w(w_stop) - self.stoppable = True + stoppable = True if num_args == 2: step = space.int_w(args_w[1]) @@ -308,7 +308,7 @@ if start < 0: raise OperationError(space.w_ValueError, space.wrap("Indicies for islice() must be non-negative integers.")) - if self.stoppable and stop < 0: + if stoppable and stop < 0: raise OperationError(space.w_ValueError, space.wrap("Stop argument must be a non-negative integer or None.")) if step < 1: raise OperationError(space.w_ValueError, space.wrap("Step must be one or lager for islice().")) @@ -321,25 +321,21 @@ return self.space.wrap(self) def next_w(self): - if self.stoppable and self.stop <= 0: - raise OperationError(self.space.w_StopIteration, self.space.w_None) - - if self.start >= 0: - skip = self.start + if self.start >= 0: # first call only + consume = self.start + 1 self.start = -1 - else: - skip = self.step - 1 - - while skip > 0: - self.space.next(self.iterable) - skip -= 1 - if self.stoppable: - self.stop -= 1 - - w_obj = self.space.next(self.iterable) - if self.stoppable: - self.stop -= 1 - return w_obj + else: # all following calls + consume = self.step + if self.stop >= 0: + if self.stop < consume: + raise OperationError(self.space.w_StopIteration, + self.space.w_None) + self.stop -= consume + while True: + w_obj = self.space.next(self.iterable) + consume -= 1 + if consume <= 0: + return w_obj def W_ISlice___new__(space, w_subtype, w_iterable, w_startstop, args_w): return space.wrap(W_ISlice(space, w_iterable, w_startstop, args_w)) Modified: pypy/dist/pypy/module/itertools/test/test_itertools.py ============================================================================== --- pypy/dist/pypy/module/itertools/test/test_itertools.py (original) +++ pypy/dist/pypy/module/itertools/test/test_itertools.py Sat Jul 19 11:33:04 2008 @@ -181,11 +181,6 @@ assert it.next() == x raises(StopIteration, it.next) - it = itertools.islice([1, 2, 3, 4, 5], 3) - for x in [1, 2, 3]: - assert it.next() == x - raises(StopIteration, it.next) - it = itertools.islice([1, 2, 3, 4, 5], 1, 3) for x in [2, 3]: assert it.next() == x @@ -201,6 +196,8 @@ assert it.next() == x raises(StopIteration, it.next) + assert list(itertools.islice(xrange(100), 10, 3)) == [] + def test_islice_overflow(self): import itertools import sys From antocuni at codespeak.net Sun Jul 20 19:00:09 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Sun, 20 Jul 2008 19:00:09 +0200 (CEST) Subject: [pypy-svn] r56693 - in pypy/branch/oo-jit/pypy/translator/cli: . test Message-ID: <20080720170009.DD6A9498098@codespeak.net> Author: antocuni Date: Sun Jul 20 19:00:08 2008 New Revision: 56693 Modified: pypy/branch/oo-jit/pypy/translator/cli/dotnet.py pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Log: use the proper attribute of _INSTANCE to get the fully qualified name, else box() will fail with instances of classes defined in external assemblies Modified: pypy/branch/oo-jit/pypy/translator/cli/dotnet.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/dotnet.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/dotnet.py Sun Jul 20 19:00:08 2008 @@ -366,7 +366,7 @@ assert isinstance(TYPE, ootype.StaticMethod) return typeof(TYPE) else: - name = '%s.%s' % (x._INSTANCE._namespace, x._INSTANCE._classname) + name = x._INSTANCE._assembly_qualified_name t = CLR.System.Type.GetType(name) assert t is not None return t Modified: pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Sun Jul 20 19:00:08 2008 @@ -585,6 +585,14 @@ return int32_type.get_Name() assert self.interpret(fn, []) == 'Int32' + def test_classof_external_assembly(self): + utils_class = classof(Utils) + def fn(): + utils_obj = box(utils_class) + utils_type = clidowncast(utils_obj, System.Type) + return utils_type.get_Name() + assert self.interpret(fn, []) == 'Utils' + def test_classof_compare(self): int32_a = classof(System.Int32) int32_b = classof(System.Int32) From antocuni at codespeak.net Mon Jul 21 14:05:33 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 21 Jul 2008 14:05:33 +0200 (CEST) Subject: [pypy-svn] r56697 - in pypy/branch/oo-jit/pypy/translator/cli: . test Message-ID: <20080721120533.4FF47169ECF@codespeak.net> Author: antocuni Date: Mon Jul 21 14:05:31 2008 New Revision: 56697 Modified: pypy/branch/oo-jit/pypy/translator/cli/metavm.py pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Log: really use the element type when storing an element of an array Modified: pypy/branch/oo-jit/pypy/translator/cli/metavm.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/metavm.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/metavm.py Mon Jul 21 14:05:31 2008 @@ -197,7 +197,7 @@ generator.ilasm.opcode('ldnull') else: generator.load(v_elem) - elemtype = generator.cts.lltype_to_cts(v_array.concretetype) + elemtype = generator.cts.lltype_to_cts(v_array.concretetype._ELEMENT) generator.ilasm.opcode('stelem', elemtype) class _TypeOf(MicroInstruction): Modified: pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Mon Jul 21 14:05:31 2008 @@ -282,6 +282,14 @@ return x[0] assert self.interpret(fn, []) is None + def test_array_setitem(self): + def fn(): + array = new_array(System.Object, 5) + array[0] = ArrayList() + return array[0].GetType().get_Name() + res = self.interpret(fn, []) + assert res == 'ArrayList' + def test_array_length(self): def fn(): x = init_array(System.Object, box(42), box(43)) From bgola at codespeak.net Mon Jul 21 15:22:35 2008 From: bgola at codespeak.net (bgola at codespeak.net) Date: Mon, 21 Jul 2008 15:22:35 +0200 (CEST) Subject: [pypy-svn] r56700 - pypy/branch/2.5-features/pypy/interpreter/pyparser/test Message-ID: <20080721132235.33C5E169EAA@codespeak.net> Author: bgola Date: Mon Jul 21 15:22:33 2008 New Revision: 56700 Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/test/test_astbuilder.py Log: tests under interpreter/pyparser/test working Modified: pypy/branch/2.5-features/pypy/interpreter/pyparser/test/test_astbuilder.py ============================================================================== --- pypy/branch/2.5-features/pypy/interpreter/pyparser/test/test_astbuilder.py (original) +++ pypy/branch/2.5-features/pypy/interpreter/pyparser/test/test_astbuilder.py Mon Jul 21 15:22:33 2008 @@ -169,9 +169,12 @@ "a[100, :2,]": "Module(None, Stmt([Discard(Subscript(Name('a'), 2, Tuple([Const(100), Sliceobj([Const(None), Const(2)])])))]))", "a[100, :]": "Module(None, Stmt([Discard(Subscript(Name('a'), 2, Tuple([Const(100), Sliceobj([Const(None), Const(None)])])))]))", - # stablecompiler can't produce AST for 2.5 syntax (yet) + # stablecompiler can't produce AST for 2.5 syntax "class A(): pass": "Module(None, Stmt([Class('A', [], None, Stmt([Pass()]))]))", + # yield changed in 2.5, so we can not trust the stablecompiler + "def f(n):\n for i in range(n):\n yield n\n": "Module(None, Stmt([Function(None, 'f', [AssName('n', 0)], [], 0, None, Stmt([For(AssName('i', 0), CallFunc(Name('range'), [Name('n')], None, None), Stmt([Discard(Yield(Name('n')))]), None)]))]))", + # stablecompiler produces a Pass statement which does not seem very consistent # (a module should only have a Stmt child) "\t # hello\n": "Module(None, Stmt([]))", From antocuni at codespeak.net Mon Jul 21 21:13:21 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 21 Jul 2008 21:13:21 +0200 (CEST) Subject: [pypy-svn] r56702 - pypy/extradoc/talk/pycon-uk-2008 Message-ID: <20080721191321.B30752A0194@codespeak.net> Author: antocuni Date: Mon Jul 21 21:13:19 2008 New Revision: 56702 Added: pypy/extradoc/talk/pycon-uk-2008/ pypy/extradoc/talk/pycon-uk-2008/abstract.txt - copied, changed from r56701, pypy/extradoc/talk/roadshow-google/abstract.txt Log: the abstract for the pycon-uk talk, mostly copied from the google roadshow one :-) From antocuni at codespeak.net Mon Jul 21 21:13:53 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 21 Jul 2008 21:13:53 +0200 (CEST) Subject: [pypy-svn] r56703 - pypy/extradoc/talk/pycon-uk-2008 Message-ID: <20080721191353.D71472A0193@codespeak.net> Author: antocuni Date: Mon Jul 21 21:13:51 2008 New Revision: 56703 Added: pypy/extradoc/talk/pycon-uk-2008/pypy-jit.txt - copied unchanged from r56702, pypy/extradoc/talk/pycon-uk-2008/abstract.txt Removed: pypy/extradoc/talk/pycon-uk-2008/abstract.txt Log: rename From cami at codespeak.net Mon Jul 21 22:00:59 2008 From: cami at codespeak.net (cami at codespeak.net) Date: Mon, 21 Jul 2008 22:00:59 +0200 (CEST) Subject: [pypy-svn] r56704 - in pypy/dist/pypy/lang/gameboy: . test Message-ID: <20080721200059.E218516852D@codespeak.net> Author: cami Date: Mon Jul 21 22:00:55 2008 New Revision: 56704 Modified: pypy/dist/pypy/lang/gameboy/cpu.py pypy/dist/pypy/lang/gameboy/test/test_cpu.py pypy/dist/pypy/lang/gameboy/test/test_cpu_2.py Log: fixed cpu_test.py: conditional test was broken, asserted pc value was wrong Modified: pypy/dist/pypy/lang/gameboy/cpu.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/cpu.py (original) +++ pypy/dist/pypy/lang/gameboy/cpu.py Mon Jul 21 22:00:55 2008 @@ -431,12 +431,13 @@ else: data = self.memory.read(self.pc.get(use_cycles)) self.pc.inc(use_cycles) # 2 cycles - # print " fetch: ", data + print " fetch: ", data return data def fetch_double_address(self): lo = self.fetch() # 1 cycle hi = self.fetch() # 1 cycle + print hex(hi), hex(lo) return (hi << 8) + lo def fetch_double_register(self, register): @@ -470,6 +471,7 @@ self.cycles += 1 def call(self, address, use_cycles=True): + print "call", hex(address), "pc", hex(self.pc.get()) # 4 cycles self.push_double_register(self.pc, use_cycles) self.pc.set(address, use_cycles=use_cycles) # 1 cycle @@ -753,7 +755,7 @@ def write_a_at_expaded_c_address(self): # LDH (C),A 2 cycles - self.write(0xFF00 + self.bc.get_lo(), self.a.get()) # 2 cycles + self.write(0xFF00 + self.c.get(), self.a.get()) # 2 cycles def load_and_increment_hli_a(self): # loadAndIncrement (HL),A 2 cycles Modified: pypy/dist/pypy/lang/gameboy/test/test_cpu.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/test/test_cpu.py (original) +++ pypy/dist/pypy/lang/gameboy/test/test_cpu.py Mon Jul 21 22:00:55 2008 @@ -138,7 +138,7 @@ cpu = get_cpu() cpu.f.set(constants.Z_FLAG) assert cpu.is_z() == True - prepare_for_fetch(cpu, 0x1234, 0x1234) + prepare_for_fetch(cpu, 0x12, 0x12) cpu.memory.write(0x1234, 0x12) assert cpu.is_z() == True cpu.rom[0x1234] = 0x12 @@ -258,9 +258,9 @@ def test_prepare_for_fetch(): cpu = get_cpu() value = 0x12 - prepare_for_fetch(cpu, value+1, value) + prepare_for_fetch(cpu, value+5, value) assert cpu.fetch() == value - assert cpu.fetch() == value+1 + assert cpu.fetch() == value+5 def prepare_for_pop(cpu, value, valueLo=None): sp = cpu.sp.get() @@ -272,9 +272,9 @@ def test_prepare_for_pop(): cpu = get_cpu() value = 0x12 - prepare_for_pop(cpu, value+1, value) + prepare_for_pop(cpu, value+5, value) assert cpu.pop() == value - assert cpu.pop() == value+1 + assert cpu.pop() == value+5 def set_registers(registers, value): #if registers is not list: @@ -298,6 +298,33 @@ prepare_for_fetch(cpu, value, value+1) assert cpu.fetch() == value+1 assert cpu.fetch() == value + +# CPU Helper Methods Testing --------------------------------------------------- + +def test_fetch_double_address(): + cpu = get_cpu() + prepare_for_fetch(cpu, 0x12, 0x34) + assert cpu.fetch_double_address() == 0x1234 + + +def test_push_double_register(): + cpu = get_cpu() + cpu.sp.set(0x03) + cpu.pc.set(0x1234) + cpu.push_double_register(cpu.pc) + assert_default_registers(cpu, sp=1, pc=0x1234) + assert cpu.pop() == 0x34 + assert cpu.pop() == 0x12 + + +def test_call(): + cpu = get_cpu() + cpu.pc.set(0x1234) + cpu.sp.set(0x03) + cpu.call(0x4321) + assert_default_registers(cpu, sp=1, pc=0x4321) + assert cpu.pop() == 0x34 + assert cpu.pop() == 0x12 # ------------------------------------------------------------------------------ # opCode Testing @@ -694,7 +721,7 @@ cycle_test(cpu, 0x27, 1) # cpl -def test_0x2F(): +def test_0x2F_complement_a(): cpu = get_cpu() value = 0x12 fValue = cpu.f.get() @@ -1167,7 +1194,7 @@ opCode += 0x08 # ldh_Ci_A -def test_0xE2(): +def test_0xE2_write_a_at_expaded_c_address(): cpu = get_cpu() value = 0x12 valueA = value+1 @@ -1243,52 +1270,58 @@ assert cpu.interrupt.is_pending() == False def conditional_call_test(cpu, opCode, flagSetter): + # set the condition to false and dont call flagSetter(cpu, False) cpu.pc.set(0) f = cpu.f.get() - cycle_test(cpu, 0xC4, 3) + cycle_test(cpu, opCode, 3) assert_default_registers(cpu, pc=2, f=f) - + # set the condition to true: unconditional_call cpu.reset() - fetchValue = 0x1234 + fetchValue = 0x4321 flagSetter(cpu, True) - cpu.sp.set(fetchValue) - prepare_for_fetch(cpu, fetchValue) + cpu.pc.set(0x1234) + cpu.sp.set(0x03) + prepare_for_fetch(cpu, fetchValue >> 8, fetchValue & 0xFF) + assert cpu.sp.get() == 0x03 f = cpu.f.get() - cycle_test(cpu, 0xC4, 6) - assert_default_registers(cpu, pc=fetchValue, sp=fetchValue-2, f=f) + cycle_test(cpu, opCode, 6) + assert_default_registers(cpu, pc=fetchValue, sp=1, f=f) + # 2 fetches happen before the pc is pushed on the stack + assert cpu.pop() == 0x34 + 2 + assert cpu.pop() == 0x12 # call_NZ_nnnn -def test_0xC4(): +def test_0xC4_conditional_call_NZ(): cpu = get_cpu() - conditional_call_test(cpu, 0xC4, setFlag0xC4) + conditional_call_test(cpu, 0xC4, set_flag_0xC4) -def setFlag0xC4(cpu, value): +def set_flag_0xC4(cpu, value): cpu.f.z_flag = not value # call_Z_nnnn -def test_0xCC(): +def test_0xCC_call_z_nnn(): cpu = get_cpu() - conditional_call_test(cpu, 0xCC, setFlag0xC4) + conditional_call_test(cpu, 0xCC, set_flag_0xCC) -def setFlag0xCC(cpu, value): - cpu.f.c_flag = not value +def set_flag_0xCC(cpu, value): + cpu.f.z_flag = value # call_NC_nnnn -def test_0xD4(): +def test_0xD4_call_nc_nnn(): cpu = get_cpu() - conditional_call_test(cpu, 0xD4, setFlag0xC4) + conditional_call_test(cpu, 0xD4, set_flag_0xD4) -def setFlag0xD4(cpu, value): - cpu.f.c_flag = value +def set_flag_0xD4(cpu, value): + cpu.f.c_flag = not value # call_C_nnnn -def test_0xDC(): +def test_0xDC_call_C_nnnn(): cpu = get_cpu() - conditional_call_test(cpu, 0xDC, setFlag0xC4) + conditional_call_test(cpu, 0xDC, set_flag_0xDC) -def setFlag0xDC(cpu, value): - cpu.f.z_flag = value +def set_flag_0xDC(cpu, value): + cpu.f.c_flag = value # push_BC to push_AF def test_0xC5_to_0xF5_push(): @@ -1394,7 +1427,7 @@ -# SECOND ORDER OPCODES --------------------------------------------------------- +# SECOND ORDER op_codes --------------------------------------------------------- def second_order_test(opCode, createFunction): cpu = get_cpu() @@ -1444,7 +1477,7 @@ second_order_test(0x38, lambda value: value >> 1) # bit_B to bit_A -def test_testBit_opCodes(): +def test_testBit_op_codes(): cpu = get_cpu() registers = [cpu.b, cpu.c, cpu.d, cpu.e, cpu.h, cpu.l, cpu.hli, cpu.a] opCode = 0x40 @@ -1469,7 +1502,7 @@ opCode += 0x01 # set_B to set_C -def test_setBit_opCodes(): +def test_setBit_op_codes(): cpu = get_cpu() registers = [cpu.b, cpu.c, cpu.d, cpu.e, cpu.h, cpu.l, cpu.hli, cpu.a] value = 0x12 @@ -1493,7 +1526,7 @@ opCode += 0x01 # res_B to res_A -def test_resetBit_opCodes(): +def test_reset_bit_op_codes(): cpu = get_cpu() registers = [cpu.b, cpu.c, cpu.d, cpu.e, cpu.h, cpu.l, cpu.hli, cpu.a] value = 0x12 Modified: pypy/dist/pypy/lang/gameboy/test/test_cpu_2.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/test/test_cpu_2.py (original) +++ pypy/dist/pypy/lang/gameboy/test/test_cpu_2.py Mon Jul 21 22:00:55 2008 @@ -85,9 +85,9 @@ def test_prepare_for_fetch(): cpu = get_cpu() value = 0x12 - prepare_for_fetch(cpu, value+1, value) + prepare_for_fetch(cpu, value+5, value) assert cpu.fetch() == value - assert cpu.fetch() == value+1 + assert cpu.fetch() == value+5 def prepare_for_pop(cpu, value, valueLo=None): sp = cpu.sp.get() @@ -99,9 +99,9 @@ def test_prepare_for_pop(): cpu = get_cpu() value = 0x12 - prepare_for_pop(cpu, value+1, value) + prepare_for_pop(cpu, value+5, value) assert cpu.pop() == value - assert cpu.pop() == value+1 + assert cpu.pop() == value+5 def set_registers(registers, value): #if registers is not list: From cami at codespeak.net Mon Jul 21 22:23:28 2008 From: cami at codespeak.net (cami at codespeak.net) Date: Mon, 21 Jul 2008 22:23:28 +0200 (CEST) Subject: [pypy-svn] r56705 - pypy/dist/pypy/lang/gameboy Message-ID: <20080721202328.57CEB169E29@codespeak.net> Author: cami Date: Mon Jul 21 22:23:26 2008 New Revision: 56705 Modified: pypy/dist/pypy/lang/gameboy/cpu.py pypy/dist/pypy/lang/gameboy/gameboy_implementation.py Log: removed false debug dependency which was only ment for local use Modified: pypy/dist/pypy/lang/gameboy/cpu.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/cpu.py (original) +++ pypy/dist/pypy/lang/gameboy/cpu.py Mon Jul 21 22:23:26 2008 @@ -2,7 +2,7 @@ from pypy.lang.gameboy import constants from pypy.lang.gameboy.ram import * from pypy.lang.gameboy.interrupt import * -from pypy.lang.gameboy.debug import * +#from pypy.lang.gameboy.debug import * # --------------------------------------------------------------------------- @@ -381,13 +381,13 @@ #print " fetch exe:", hex(opCode), " " #, FETCH_EXECUTE_OP_CODES[opCode].__name__ self.last_fetch_execute_op_code = opCode - if DEBUG: log(opCode, is_fetch_execute=True) + #if DEBUG: log(opCode, is_fetch_execute=True) FETCH_EXECUTE_OP_CODES[opCode](self) def execute(self, opCode): self.instruction_counter += 1 - if DEBUG: log(opCode) + #if DEBUG: log(opCode) #print self.instruction_counter, "-"*60 #print "exe: ", hex(opCode), " " #, OP_CODES[opCode].__name__ @@ -431,13 +431,11 @@ else: data = self.memory.read(self.pc.get(use_cycles)) self.pc.inc(use_cycles) # 2 cycles - print " fetch: ", data return data def fetch_double_address(self): lo = self.fetch() # 1 cycle hi = self.fetch() # 1 cycle - print hex(hi), hex(lo) return (hi << 8) + lo def fetch_double_register(self, register): @@ -471,7 +469,6 @@ self.cycles += 1 def call(self, address, use_cycles=True): - print "call", hex(address), "pc", hex(self.pc.get()) # 4 cycles self.push_double_register(self.pc, use_cycles) self.pc.set(address, use_cycles=use_cycles) # 1 cycle Modified: pypy/dist/pypy/lang/gameboy/gameboy_implementation.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/gameboy_implementation.py (original) +++ pypy/dist/pypy/lang/gameboy/gameboy_implementation.py Mon Jul 21 22:23:26 2008 @@ -6,7 +6,7 @@ from pypy.lang.gameboy.sound import SoundDriver from pypy.lang.gameboy.timer import Clock from pypy.lang.gameboy import constants -from pypy.lang.gameboy import debug +#from pypy.lang.gameboy import debug from pypy.rlib.rsdl import RSDL, RSDL_helper from pypy.rpython.lltypesystem import lltype, rffi @@ -41,7 +41,7 @@ finally: lltype.free(self.event, flavor='raw') RSDL.Quit() - debug.print_results() + #debug.print_results() return 0 def handle_events(self): @@ -93,7 +93,7 @@ #if y%2 == 0 or True: # px = self.get_pixel_color(x, y) # str += ["#", "%", "+", " ", " "][px] - #RSDL_helper.set_pixel(self.screen, x, y, self.pixel_map(x, y)) + RSDL_helper.set_pixel(self.screen, x, y, self.pixel_map(x, y)) pass #print str; From antocuni at codespeak.net Tue Jul 22 13:53:40 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 22 Jul 2008 13:53:40 +0200 (CEST) Subject: [pypy-svn] r56709 - in pypy/branch/oo-jit/pypy: jit/codegen/cli jit/codegen/cli/test translator/cli/src Message-ID: <20080722115340.C816F169DFB@codespeak.net> Author: antocuni Date: Tue Jul 22 13:53:38 2008 New Revision: 56709 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Log: (in-progress) start implementing flexswitches for the CLI backend. Not finished yet. Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Tue Jul 22 13:53:38 2008 @@ -194,6 +194,64 @@ self.gv_value.load(self.builder) self.builder.graphbuilder.il.Emit(OpCodes.Stfld, self.fieldinfo) +class DoFlexSwitch(Operation): + + def __init__(self, builder, gv_flexswitch, gv_exitswitch, args_gv): + self.builder = builder + self.gv_flexswitch = gv_flexswitch + self.gv_exitswitch = gv_exitswitch + self.args_gv = args_gv # XXX: remove duplicates + + def restype(self): + return None + + def emit(self): + gbuilder = self.builder.graphbuilder + il = gbuilder.il + # get MethodInfo for LowLevelFlexSwitch.execute + clitype = self.gv_flexswitch.flexswitch.GetType() + meth_execute = clitype.GetMethod('execute') + + # setup the correct inputargs + manager = InputArgsManager(gbuilder, self.args_gv) + manager.copy_from_args(self.builder) + + # jumpto = flexswitch.execute(exitswitch, inputargs); + # goto dispatch_jump; + self.gv_flexswitch.load(self.builder) + self.gv_exitswitch.load(self.builder) + il.Emit(OpCodes.Ldloc, gbuilder.inputargs_var) + il.Emit(OpCodes.Callvirt, meth_execute) + il.Emit(OpCodes.Stloc, gbuilder.jumpto_var) + il.Emit(OpCodes.Br, gbuilder.dispatch_jump_label) + + +class InputArgsManager: + + def __init__(self, graphbuilder, args_gv): + self.inputargs_var = graphbuilder.inputargs_var + self.inputargs_clitype = graphbuilder.inputargs_clitype + self.args_gv = args_gv + + def basename_from_type(self, clitype): + return clitype.get_Name() + + def copy_from_args(self, builder): + il = builder.graphbuilder.il + inputargs_var = self.inputargs_var + inputargs_clitype = self.inputargs_clitype + counters = {} + for gv_arg in self.args_gv: + clitype = gv_arg.getCliType() + basename = self.basename_from_type(clitype) + count = counters.get(clitype, 0) + fieldname = '%s_%d' % (basename, count) + counters[clitype] = count+1 + field = inputargs_clitype.GetField(fieldname) + + il.Emit(OpCodes.Ldloc, inputargs_var) + gv_arg.load(builder) + il.Emit(OpCodes.Stfld, field) class WriteLine(Operation): Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Tue Jul 22 13:53:38 2008 @@ -4,13 +4,17 @@ from pypy.rpython.lltypesystem import lltype from pypy.rlib.objectmodel import specialize from pypy.jit.codegen.model import AbstractRGenOp, GenBuilder, GenLabel -from pypy.jit.codegen.model import GenVarOrConst, GenVar, GenConst, CodeGenSwitch +from pypy.jit.codegen.model import GenVarOrConst, GenVar, GenConst +from pypy.jit.codegen.model import CodeGenSwitch from pypy.jit.codegen.cli import operation as ops from pypy.jit.codegen.cli.methodbuilder import get_methodbuilder -from pypy.translator.cli.dotnet import CLR, typeof, new_array, box, unbox, clidowncast, classof +from pypy.translator.cli.dotnet import CLR, typeof, new_array, init_array +from pypy.translator.cli.dotnet import box, unbox, clidowncast, classof from pypy.translator.cli import dotnet System = CLR.System DelegateHolder = CLR.pypy.runtime.DelegateHolder +LowLevelFlexSwitch = CLR.pypy.runtime.LowLevelFlexSwitch +InputArgs = CLR.pypy.runtime.InputArgs OpCodes = System.Reflection.Emit.OpCodes cVoid = ootype.nullruntimeclass @@ -20,6 +24,8 @@ cObject = classof(System.Object) cString = classof(System.String) cChar = classof(System.Char) +cInputArgs = classof(InputArgs) +cUtils = classof(CLR.pypy.runtime.Utils) class SigToken: def __init__(self, args, res, funcclass): @@ -213,6 +219,19 @@ assert T is ootype.Object return ootype.cast_to_object(self.holder.GetFunc()) +class FlexSwitchConst(BaseConst): + + def __init__(self, flexswitch): + self.flexswitch = flexswitch + + def getobj(self): + return self.flexswitch + + def load(self, builder): + index = self._get_index(builder) + self._load_from_array(builder, index, self.flexswitch.GetType()) + + class Label(GenLabel): def __init__(self, label, inputargs_gv): self.label = label @@ -338,6 +357,8 @@ else: self.retvar = None self.retlabel = self.il.DefineLabel() + self.blockids = {} + self.has_flexswitches = False def appendbranch(self, branchbuilder): self.branches.append(branchbuilder) @@ -395,11 +416,20 @@ def genop_ooisnull(self, gv_obj): raise NotImplementedError + def flexswitch(self, gv_exitswitch, args_gv): + return self.branches[0].flexswitch(gv_exitswitch, args_gv) + def end(self): + # emit initialization code + self.emit_preamble() + # render all the pending branches for branchbuilder in self.branches: branchbuilder.replayops() + # emit dispatch_jump, if there are flexswitches + self.emit_dispatch_jump() + # render the return block for last, else the verifier could complain self.il.MarkLabel(self.retlabel) if self.retvar: @@ -414,6 +444,45 @@ myfunc = self.meth.create_delegate(self.delegatetype, consts) self.gv_entrypoint.holder.SetFunc(myfunc) + def _setup_flexswitches(self): + if self.has_flexswitches: + return + self.has_flexswitches = True + self.dispatch_jump_label = self.il.DefineLabel() + self.inputargs_clitype = class2type(cInputArgs) + self.inputargs_var = self.il.DeclareLocal(self.inputargs_clitype) + self.jumpto_var = self.il.DeclareLocal(class2type(cInt32)) + + def emit_preamble(self): + if not self.has_flexswitches: + return + # InputArgs inputargs_var = new InputArgs() + clitype = class2type(cInputArgs) + ctor = clitype.GetConstructor(new_array(System.Type, 0)) + self.il.Emit(OpCodes.Newobj, ctor) + self.il.Emit(OpCodes.Stloc, self.inputargs_var) + + def emit_dispatch_jump(self): + if not self.has_flexswitches: + return + # make sure we don't enter dispatch_jump by mistake + self.il.Emit(OpCodes.Br, self.retlabel) + self.il.MarkLabel(self.dispatch_jump_label) + + labels = new_array(System.Reflection.Emit.Label, + len(self.blockids)) + for genlabel, blockid in self.blockids.iteritems(): + labels[blockid] = genlabel.label + + self.il.Emit(OpCodes.Ldloc, self.jumpto_var) + self.il.Emit(OpCodes.Switch, labels) + # XXX: handle blockids that are inside flexswitch cases + # default: Utils.throwInvalidBlockId(jumpto) + clitype = class2type(cUtils) + meth = clitype.GetMethod("throwInvalidBlockId") + self.il.Emit(OpCodes.Ldloc, self.jumpto_var) + self.il.Emit(OpCodes.Call, meth) + class BranchBuilder(GenBuilder): @@ -488,7 +557,13 @@ seen[gv] = None label = self.graphbuilder.il.DefineLabel() self.appendop(ops.MarkLabel(self, label)) - return Label(label, args_gv) + return self.create_label(label, args_gv) + + def create_label(self, label, args_gv): + res = Label(label, args_gv) + blockids = self.graphbuilder.blockids + blockids[res] = len(blockids) + return res def _jump_if(self, gv_condition, opcode): label = self.graphbuilder.il.DefineLabel() @@ -504,6 +579,22 @@ def jump_if_true(self, gv_condition, args_for_jump_gv): return self._jump_if(gv_condition, OpCodes.Brtrue) + def flexswitch(self, gv_exitswitch, args_gv): + self.graphbuilder._setup_flexswitches() + flexswitch = IntFlexSwitch() + flexswitch.xxxbuilder = BranchBuilder(self.graphbuilder, self.graphbuilder.il.DefineLabel()) + gv_flexswitch = flexswitch.gv_flexswitch + lbl = self.graphbuilder.il.DefineLabel() + default_label = self.create_label(lbl, args_gv) + default_blockid = self.graphbuilder.blockids[default_label] + flexswitch.llflexswitch.set_default_blockid(default_blockid) + op = ops.DoFlexSwitch(self, gv_flexswitch, gv_exitswitch, args_gv) + self.appendop(op) + default_builder = BranchBuilder(self.graphbuilder, default_label.label) + self.graphbuilder.appendbranch(default_builder) + self.is_open = False + return flexswitch, default_builder + def appendop(self, op): self.operations.append(op) @@ -517,6 +608,19 @@ for op in self.operations: op.emit() + +class IntFlexSwitch(CodeGenSwitch): + + def __init__(self): + self.llflexswitch = LowLevelFlexSwitch() + self.gv_flexswitch = FlexSwitchConst(self.llflexswitch) + + def add_case(self, gv_case): + return self.xxxbuilder + #import pdb;pdb.set_trace() + + + global_rgenop = RCliGenOp() RCliGenOp.constPrebuiltGlobal = global_rgenop.genconst zero_const = ObjectConst(ootype.NULL) Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Tue Jul 22 13:53:38 2008 @@ -34,7 +34,7 @@ py.test.skip('fixme') def test_switch_direct(self): - py.test.skip('no promotion/flexswitch for now please :-)') + py.test.skip('in-progress') def test_large_switch_direct(self): py.test.skip('no promotion/flexswitch for now please :-)') Modified: pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs (original) +++ pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Tue Jul 22 13:53:38 2008 @@ -103,6 +103,43 @@ namespace pypy.runtime { + // XXX: these classes should be automatically generated by the JIT backend + public delegate int FlexSwitchCase(int block, InputArgs args); + public class InputArgs { + public int Int32_0; + public int Int32_1; + } + + public class LowLevelFlexSwitch + { + public int default_blockid = -1; + public int numcases = 0; + public int[] values = new int[10]; // XXX maxlength? + public FlexSwitchCase[] cases = new FlexSwitchCase[10]; + + public void set_default_blockid(int blockid) + { + this.default_blockid = blockid; + } + + public void add_case(int v, FlexSwitchCase c) + { + values[numcases] = v; + cases[numcases] = c; + numcases++; + } + + public int execute(int v, InputArgs args) + { + for(int i=0; i Author: antocuni Date: Tue Jul 22 17:24:11 2008 New Revision: 56713 Modified: pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Log: make sure we can use valuetypes as dictionay keys Modified: pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Tue Jul 22 17:24:11 2008 @@ -585,6 +585,13 @@ return f self.interpret(fn, []) + def test_valuetype_key(self): + def fn(): + d = {} + d[OpCodes.Add] = 42 + return d[OpCodes.Add] + assert self.interpret(fn, []) == 42 + def test_classof(self): int32_class = classof(System.Int32) def fn(): From antocuni at codespeak.net Tue Jul 22 17:53:13 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 22 Jul 2008 17:53:13 +0200 (CEST) Subject: [pypy-svn] r56714 - in pypy/branch/oo-jit/pypy: jit/codegen/cli/test translator/cli translator/cli/src Message-ID: <20080722155313.AE313169E77@codespeak.net> Author: antocuni Date: Tue Jul 22 17:53:13 2008 New Revision: 56714 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_interpreter.py pypy/branch/oo-jit/pypy/translator/cli/dotnet.py pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Log: add two known delegate types, and skip some tests Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_interpreter.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_interpreter.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_interpreter.py Tue Jul 22 17:53:13 2008 @@ -99,3 +99,8 @@ test_degenerated_at_return = skip test_degenerated_via_substructure = skip test_red_subclass = skip + + test_builtin_oosend_with_green_args = skip + test_red_isinstance_degenerated = skip + test_red_array = skip + Modified: pypy/branch/oo-jit/pypy/translator/cli/dotnet.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/dotnet.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/dotnet.py Tue Jul 22 17:53:13 2008 @@ -690,6 +690,8 @@ ootype.StaticMethod([], ootype.Signed): CLR.pypy.test.DelegateType_int__0, ootype.StaticMethod([ootype.Signed, ootype.Float], ootype.Float): CLR.pypy.test.DelegateType_double_int_double, ootype.StaticMethod([ootype.Float], ootype.Float): CLR.pypy.test.DelegateType_double__double_1, + ootype.StaticMethod([ootype.Bool], ootype.Bool): CLR.pypy.test.DelegateType_bool_bool_1, + ootype.StaticMethod([ootype.Char], ootype.Char): CLR.pypy.test.DelegateType_char_char_1, ootype.StaticMethod([ootype.Signed], ootype.Void): CLR.pypy.test.DelegateType_void_int_1, ootype.StaticMethod([ootype.Signed], ootype.Signed): CLR.pypy.test.DelegateType_int__int_1, ootype.StaticMethod([ootype.Signed] * 2, ootype.Signed): CLR.pypy.test.DelegateType_int__int_2, Modified: pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs (original) +++ pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Tue Jul 22 17:53:13 2008 @@ -87,6 +87,8 @@ public delegate double DelegateType_double__double_1(double a); public delegate double DelegateType_double_int_double(int a, double b); public delegate void DelegateType_void_int_1(int a); + public delegate bool DelegateType_bool_bool_1(bool a); + public delegate char DelegateType_char_char_1(char a); public delegate int DelegateType_int__int_1(int a); public delegate int DelegateType_int__int_2(int a, int b); public delegate int DelegateType_int__int_3(int a, int b, int c); From antocuni at codespeak.net Tue Jul 22 23:31:42 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 22 Jul 2008 23:31:42 +0200 (CEST) Subject: [pypy-svn] r56724 - pypy/branch/oo-jit/pypy/jit/codegen/cli Message-ID: <20080722213142.A36BC16856E@codespeak.net> Author: antocuni Date: Tue Jul 22 23:31:41 2008 New Revision: 56724 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Log: a bit of refactoring of the cli backend: - most of GraphBuilder has been moved to a new parent class MethodBuilder; a MethodBuilder is a builder which generates a delegates; this may not correspond to a whole graph when there are flexswitches - informations about the whole graph, such as blockids and the presence of flexswitches, are stored in a separate GraphInfo class, which is shared by every builder More refactoring is coming Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Tue Jul 22 23:31:41 2008 @@ -233,7 +233,8 @@ class Label(GenLabel): - def __init__(self, label, inputargs_gv): + def __init__(self, blockid, label, inputargs_gv): + self.blockid = blockid self.label = label self.inputargs_gv = inputargs_gv @@ -332,33 +333,37 @@ args[0] = System.Type.GetType("System.Object[]") for i in range(len(argsclass)): args[i+1] = class2type(argsclass[i]) - res = class2type(sigtoken.res) - builder = GraphBuilder(self, name, res, args, sigtoken) + restype = class2type(sigtoken.res) + delegatetype = class2type(sigtoken.funcclass) + builder = GraphBuilder(self, name, restype, args, delegatetype) return builder, builder.gv_entrypoint, builder.inputargs_gv[:] -class GraphBuilder(GenBuilder): +class GraphInfo: + def __init__(self): + self.has_flexswitches = False + self.blocks = [] # blockid -> (methodbuilder, label) - def __init__(self, rgenop, name, res, args, sigtoken): +class MethodBuilder(GenBuilder): + + def __init__(self, rgenop, name, restype, args, delegatetype, graphinfo): self.rgenop = rgenop - self.meth = get_methodbuilder(name, res, args) + self.meth = get_methodbuilder(name, restype, args) self.il = self.meth.get_il_generator() self.inputargs_gv = [] # we start from 1 because the 1st arg is an Object[] containing the genconsts for i in range(1, len(args)): self.inputargs_gv.append(GenArgVar(i, args[i])) - self.delegatetype = class2type(sigtoken.funcclass) - self.gv_entrypoint = FunctionConst(self.delegatetype) + self.delegatetype = delegatetype + self.gv_entrypoint = FunctionConst(delegatetype) self.genconsts = {} self.branches = [BranchBuilder(self, self.il.DefineLabel())] - restype = class2type(sigtoken.res) if restype is not None: self.retvar = self.il.DeclareLocal(restype) else: self.retvar = None self.retlabel = self.il.DefineLabel() - self.blockids = {} - self.has_flexswitches = False + self.graphinfo = graphinfo def appendbranch(self, branchbuilder): self.branches.append(branchbuilder) @@ -428,7 +433,7 @@ branchbuilder.replayops() # emit dispatch_jump, if there are flexswitches - self.emit_dispatch_jump() + self.emit_before_returnblock() # render the return block for last, else the verifier could complain self.il.MarkLabel(self.retlabel) @@ -444,17 +449,29 @@ myfunc = self.meth.create_delegate(self.delegatetype, consts) self.gv_entrypoint.holder.SetFunc(myfunc) - def _setup_flexswitches(self): - if self.has_flexswitches: + def emit_preamble(self): + pass + + def emit_before_returnblock(self): + pass + + +class GraphBuilder(MethodBuilder): + def __init__(self, rgenop, name, restype, args, delegatetype): + graphinfo = GraphInfo() + MethodBuilder.__init__(self, rgenop, name, restype, args, delegatetype, graphinfo) + + def setup_flexswitches(self): + if self.graphinfo.has_flexswitches: return - self.has_flexswitches = True + self.graphinfo.has_flexswitches = True self.dispatch_jump_label = self.il.DefineLabel() self.inputargs_clitype = class2type(cInputArgs) self.inputargs_var = self.il.DeclareLocal(self.inputargs_clitype) self.jumpto_var = self.il.DeclareLocal(class2type(cInt32)) def emit_preamble(self): - if not self.has_flexswitches: + if not self.graphinfo.has_flexswitches: return # InputArgs inputargs_var = new InputArgs() clitype = class2type(cInputArgs) @@ -462,16 +479,17 @@ self.il.Emit(OpCodes.Newobj, ctor) self.il.Emit(OpCodes.Stloc, self.inputargs_var) - def emit_dispatch_jump(self): - if not self.has_flexswitches: + def emit_before_returnblock(self): + if not self.graphinfo.has_flexswitches: return # make sure we don't enter dispatch_jump by mistake self.il.Emit(OpCodes.Br, self.retlabel) self.il.MarkLabel(self.dispatch_jump_label) labels = new_array(System.Reflection.Emit.Label, - len(self.blockids)) - for genlabel, blockid in self.blockids.iteritems(): + len(self.graphinfo.blocks)) + for blockid, (builder, genlabel) in self.graphinfo.blocks: + assert builder is self labels[blockid] = genlabel.label self.il.Emit(OpCodes.Ldloc, self.jumpto_var) @@ -484,6 +502,8 @@ self.il.Emit(OpCodes.Call, meth) + + class BranchBuilder(GenBuilder): def __init__(self, graphbuilder, label): @@ -560,10 +580,11 @@ return self.create_label(label, args_gv) def create_label(self, label, args_gv): - res = Label(label, args_gv) - blockids = self.graphbuilder.blockids - blockids[res] = len(blockids) - return res + blocks = self.graphbuilder.graphinfo.blocks + blockid = len(blocks) + result = Label(blockid, label, args_gv) + blocks.append((self.graphbuilder, result)) + return result def _jump_if(self, gv_condition, opcode): label = self.graphbuilder.il.DefineLabel() @@ -580,14 +601,14 @@ return self._jump_if(gv_condition, OpCodes.Brtrue) def flexswitch(self, gv_exitswitch, args_gv): - self.graphbuilder._setup_flexswitches() + # XXX: this code is valid only for GraphBuilder + self.graphbuilder.setup_flexswitches() flexswitch = IntFlexSwitch() flexswitch.xxxbuilder = BranchBuilder(self.graphbuilder, self.graphbuilder.il.DefineLabel()) gv_flexswitch = flexswitch.gv_flexswitch lbl = self.graphbuilder.il.DefineLabel() default_label = self.create_label(lbl, args_gv) - default_blockid = self.graphbuilder.blockids[default_label] - flexswitch.llflexswitch.set_default_blockid(default_blockid) + flexswitch.llflexswitch.set_default_blockid(default_label.blockid) op = ops.DoFlexSwitch(self, gv_flexswitch, gv_exitswitch, args_gv) self.appendop(op) default_builder = BranchBuilder(self.graphbuilder, default_label.label) From antocuni at codespeak.net Tue Jul 22 23:50:14 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 22 Jul 2008 23:50:14 +0200 (CEST) Subject: [pypy-svn] r56725 - pypy/branch/oo-jit/pypy/jit/codegen/cli Message-ID: <20080722215014.6A575169EC6@codespeak.net> Author: antocuni Date: Tue Jul 22 23:50:13 2008 New Revision: 56725 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Log: more refactoring: rename graphbuilder to methbuilder everywhere; moreover, we now pass directly the methbuilder when constructing Operations or when loading/storing a GenVar, instead of passing a branchbuilder and getting the methbuilder from it Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Tue Jul 22 23:50:13 2008 @@ -17,7 +17,7 @@ if self._gv_res is None: restype = self.restype() if restype is not None: - loc = self.builder.graphbuilder.il.DeclareLocal(restype) + loc = self.methbuilder.il.DeclareLocal(restype) self._gv_res = GenLocalVar(loc) return self._gv_res @@ -28,38 +28,38 @@ raise NotImplementedError def storeResult(self): - self.gv_res().store(self.builder) + self.gv_res().store(self.methbuilder) class UnaryOp(Operation): - def __init__(self, builder, gv_x): - self.builder = builder + def __init__(self, methbuilder, gv_x): + self.methbuilder = methbuilder self.gv_x = gv_x def pushAllArgs(self): - self.gv_x.load(self.builder) + self.gv_x.load(self.methbuilder) def emit(self): self.pushAllArgs() - self.builder.graphbuilder.il.Emit(self.getOpCode()) + self.methbuilder.il.Emit(self.getOpCode()) self.storeResult() def getOpCode(self): raise NotImplementedError class BinaryOp(Operation): - def __init__(self, builder, gv_x, gv_y): - self.builder = builder + def __init__(self, methbuilder, gv_x, gv_y): + self.methbuilder = methbuilder self.gv_x = gv_x self.gv_y = gv_y def pushAllArgs(self): - self.gv_x.load(self.builder) - self.gv_y.load(self.builder) + self.gv_x.load(self.methbuilder) + self.gv_y.load(self.methbuilder) def emit(self): self.pushAllArgs() - self.builder.graphbuilder.il.Emit(self.getOpCode()) + self.methbuilder.il.Emit(self.getOpCode()) self.storeResult() def getOpCode(self): @@ -69,25 +69,25 @@ class SameAs(UnaryOp): def emit(self): gv_res = self.gv_res() - self.gv_x.load(self.builder) - self.gv_res().store(self.builder) + self.gv_x.load(self.methbuilder) + self.gv_res().store(self.methbuilder) class MarkLabel(Operation): - def __init__(self, builder, label): - self.builder = builder + def __init__(self, methbuilder, label): + self.methbuilder = methbuilder self.label = label def restype(self): return None def emit(self): - self.builder.graphbuilder.il.MarkLabel(self.label) + self.methbuilder.il.MarkLabel(self.label) class FollowLink(Operation): - def __init__(self, builder, outputargs_gv, inputargs_gv, label): - self.builder = builder + def __init__(self, methbuilder, outputargs_gv, inputargs_gv, label): + self.methbuilder = methbuilder self.outputargs_gv = outputargs_gv self.inputargs_gv = inputargs_gv self.label = label @@ -97,15 +97,15 @@ def emit(self): for i in range(len(self.outputargs_gv)): - self.outputargs_gv[i].load(self.builder) - self.inputargs_gv[i].store(self.builder) - self.builder.graphbuilder.il.Emit(OpCodes.Br, self.label) + self.outputargs_gv[i].load(self.methbuilder) + self.inputargs_gv[i].store(self.methbuilder) + self.methbuilder.il.Emit(OpCodes.Br, self.label) class Branch(Operation): - def __init__(self, builder, gv_cond, opcode, label): - self.builder = builder + def __init__(self, methbuilder, gv_cond, opcode, label): + self.methbuilder = methbuilder self.gv_cond = gv_cond self.opcode = opcode self.label = label @@ -115,31 +115,31 @@ def emit(self): if self.gv_cond is not None: - self.gv_cond.load(self.builder) - self.builder.graphbuilder.il.Emit(self.opcode, self.label) + self.gv_cond.load(self.methbuilder) + self.methbuilder.il.Emit(self.opcode, self.label) class Return(Operation): - def __init__(self, builder, gv_x): - self.builder = builder + def __init__(self, methbuilder, gv_x): + self.methbuilder = methbuilder self.gv_x = gv_x def restype(self): return None def emit(self): - retvar = self.builder.graphbuilder.retvar - retlabel = self.builder.graphbuilder.retlabel + retvar = self.methbuilder.retvar + retlabel = self.methbuilder.retlabel if self.gv_x is not None: - self.gv_x.load(self.builder) - self.builder.graphbuilder.il.Emit(OpCodes.Stloc, retvar) - self.builder.graphbuilder.il.Emit(OpCodes.Br, retlabel) + self.gv_x.load(self.methbuilder) + self.methbuilder.il.Emit(OpCodes.Stloc, retvar) + self.methbuilder.il.Emit(OpCodes.Br, retlabel) class Call(Operation): - def __init__(self, builder, sigtoken, gv_fnptr, args_gv): + def __init__(self, methbuilder, sigtoken, gv_fnptr, args_gv): from pypy.jit.codegen.cli.rgenop import class2type - self.builder = builder + self.methbuilder = methbuilder self.sigtoken = sigtoken self.gv_fnptr = gv_fnptr self.args_gv = args_gv @@ -152,18 +152,18 @@ from pypy.jit.codegen.cli.rgenop import class2type delegate_type = class2type(self.sigtoken.funcclass) meth_invoke = delegate_type.GetMethod('Invoke') - self.gv_fnptr.load(self.builder) - self.builder.graphbuilder.il.Emit(OpCodes.Castclass, delegate_type) + self.gv_fnptr.load(self.methbuilder) + self.methbuilder.il.Emit(OpCodes.Castclass, delegate_type) for gv_arg in self.args_gv: - gv_arg.load(self.builder) - self.builder.graphbuilder.il.EmitCall(OpCodes.Callvirt, meth_invoke, None) + gv_arg.load(self.methbuilder) + self.methbuilder.il.EmitCall(OpCodes.Callvirt, meth_invoke, None) self.storeResult() class GetField(Operation): - def __init__(self, builder, gv_obj, fieldname): - self.builder = builder + def __init__(self, methbuilder, gv_obj, fieldname): + self.methbuilder = methbuilder self.gv_obj = gv_obj clitype = gv_obj.getCliType() self.fieldinfo = clitype.GetField(fieldname) @@ -172,15 +172,15 @@ return self.fieldinfo.get_FieldType() def emit(self): - self.gv_obj.load(self.builder) - self.builder.graphbuilder.il.Emit(OpCodes.Ldfld, self.fieldinfo) + self.gv_obj.load(self.methbuilder) + self.methbuilder.il.Emit(OpCodes.Ldfld, self.fieldinfo) self.storeResult() class SetField(Operation): - def __init__(self, builder, gv_obj, gv_value, fieldname): - self.builder = builder + def __init__(self, methbuilder, gv_obj, gv_value, fieldname): + self.methbuilder = methbuilder self.gv_obj = gv_obj self.gv_value = gv_value clitype = gv_obj.getCliType() @@ -190,14 +190,14 @@ return None def emit(self): - self.gv_obj.load(self.builder) - self.gv_value.load(self.builder) - self.builder.graphbuilder.il.Emit(OpCodes.Stfld, self.fieldinfo) + self.gv_obj.load(self.methbuilder) + self.gv_value.load(self.methbuilder) + self.methbuilder.il.Emit(OpCodes.Stfld, self.fieldinfo) class DoFlexSwitch(Operation): - def __init__(self, builder, gv_flexswitch, gv_exitswitch, args_gv): - self.builder = builder + def __init__(self, methbuilder, gv_flexswitch, gv_exitswitch, args_gv): + self.methbuilder = methbuilder self.gv_flexswitch = gv_flexswitch self.gv_exitswitch = gv_exitswitch self.args_gv = args_gv # XXX: remove duplicates @@ -206,38 +206,38 @@ return None def emit(self): - gbuilder = self.builder.graphbuilder - il = gbuilder.il + mbuilder = self.methbuilder + il = mbuilder.il # get MethodInfo for LowLevelFlexSwitch.execute clitype = self.gv_flexswitch.flexswitch.GetType() meth_execute = clitype.GetMethod('execute') # setup the correct inputargs - manager = InputArgsManager(gbuilder, self.args_gv) - manager.copy_from_args(self.builder) + manager = InputArgsManager(mbuilder, self.args_gv) + manager.copy_from_args(mbuilder) # jumpto = flexswitch.execute(exitswitch, inputargs); # goto dispatch_jump; - self.gv_flexswitch.load(self.builder) - self.gv_exitswitch.load(self.builder) - il.Emit(OpCodes.Ldloc, gbuilder.inputargs_var) + self.gv_flexswitch.load(mbuilder) + self.gv_exitswitch.load(mbuilder) + il.Emit(OpCodes.Ldloc, mbuilder.inputargs_var) il.Emit(OpCodes.Callvirt, meth_execute) - il.Emit(OpCodes.Stloc, gbuilder.jumpto_var) - il.Emit(OpCodes.Br, gbuilder.dispatch_jump_label) + il.Emit(OpCodes.Stloc, mbuilder.jumpto_var) + il.Emit(OpCodes.Br, mbuilder.dispatch_jump_label) class InputArgsManager: - def __init__(self, graphbuilder, args_gv): - self.inputargs_var = graphbuilder.inputargs_var - self.inputargs_clitype = graphbuilder.inputargs_clitype + def __init__(self, methbuilder, args_gv): + self.inputargs_var = methbuilder.inputargs_var + self.inputargs_clitype = methbuilder.inputargs_clitype self.args_gv = args_gv def basename_from_type(self, clitype): return clitype.get_Name() - def copy_from_args(self, builder): - il = builder.graphbuilder.il + def copy_from_args(self, methbuilder): + il = methbuilder.methbuilder.il inputargs_var = self.inputargs_var inputargs_clitype = self.inputargs_clitype counters = {} @@ -255,15 +255,15 @@ class WriteLine(Operation): - def __init__(self, builder, message): - self.builder = builder + def __init__(self, methbuilder, message): + self.methbuilder = methbuilder self.message = message def restype(self): return None def emit(self): - self.builder.graphbuilder.il.EmitWriteLine(self.message) + self.methbuilder.il.EmitWriteLine(self.message) def opcode2attrname(opcode): if opcode == 'ldc.r8 0': @@ -327,7 +327,7 @@ if 'call' in step: return # XXX, fix this attrname = opcode2attrname(step) - body.append('self.builder.graphbuilder.il.Emit(OpCodes.%s)' % attrname) + body.append('self.methbuilder.il.Emit(OpCodes.%s)' % attrname) elif isinstance(step, cli_opcodes.MapException): return # XXX, TODO else: Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Tue Jul 22 23:50:13 2008 @@ -60,20 +60,20 @@ def getCliType(self): return self.cliType - def load(self, builder): + def load(self, methbuilder): if self.index == 0: - builder.graphbuilder.il.Emit(OpCodes.Ldarg_0) + methbuilder.il.Emit(OpCodes.Ldarg_0) elif self.index == 1: - builder.graphbuilder.il.Emit(OpCodes.Ldarg_1) + methbuilder.il.Emit(OpCodes.Ldarg_1) elif self.index == 2: - builder.graphbuilder.il.Emit(OpCodes.Ldarg_2) + methbuilder.il.Emit(OpCodes.Ldarg_2) elif self.index == 3: - builder.graphbuilder.il.Emit(OpCodes.Ldarg_3) + methbuilder.il.Emit(OpCodes.Ldarg_3) else: - builder.graphbuilder.il.Emit(OpCodes.Ldarg, self.index) + methbuilder.il.Emit(OpCodes.Ldarg, self.index) - def store(self, builder): - builder.graphbuilder.il.Emit(OpCodes.Starg, self.index) + def store(self, methbuilder): + methbuilder.il.Emit(OpCodes.Starg, self.index) def __repr__(self): return "GenArgVar(%d)" % self.index @@ -85,11 +85,11 @@ def getCliType(self): return self.v.get_LocalType() - def load(self, builder): - builder.graphbuilder.il.Emit(OpCodes.Ldloc, self.v) + def load(self, methbuilder): + methbuilder.il.Emit(OpCodes.Ldloc, self.v) - def store(self, builder): - builder.graphbuilder.il.Emit(OpCodes.Stloc, self.v) + def store(self, methbuilder): + methbuilder.il.Emit(OpCodes.Stloc, self.v) class IntConst(GenConst): @@ -109,8 +109,8 @@ def getCliType(self): return class2type(self.cliclass) - def load(self, builder): - builder.graphbuilder.il.Emit(OpCodes.Ldc_I4, self.value) + def load(self, methbuilder): + methbuilder.il.Emit(OpCodes.Ldc_I4, self.value) def __repr__(self): return "int const=%s" % self.value @@ -130,28 +130,28 @@ def getCliType(self): return typeof(System.Double) - def load(self, builder): - builder.graphbuilder.il.Emit(OpCodes.Ldc_R8, self.value) + def load(self, methbuilder): + methbuilder.il.Emit(OpCodes.Ldc_R8, self.value) def __repr__(self): return "float const=%s" % self.value class BaseConst(GenConst): - def _get_index(self, builder): + def _get_index(self, methbuilder): # check whether there is already an index associated to this const try: - index = builder.genconsts[self] + index = methbuilder.genconsts[self] except KeyError: - index = len(builder.genconsts) - builder.genconsts[self] = index + index = len(methbuilder.genconsts) + methbuilder.genconsts[self] = index return index - def _load_from_array(self, builder, index, clitype): - builder.graphbuilder.il.Emit(OpCodes.Ldarg_0) - builder.graphbuilder.il.Emit(OpCodes.Ldc_I4, index) - builder.graphbuilder.il.Emit(OpCodes.Ldelem_Ref) - builder.graphbuilder.il.Emit(OpCodes.Castclass, clitype) + def _load_from_array(self, methbuilder, index, clitype): + methbuilder.il.Emit(OpCodes.Ldarg_0) + methbuilder.il.Emit(OpCodes.Ldc_I4, index) + methbuilder.il.Emit(OpCodes.Ldelem_Ref) + methbuilder.il.Emit(OpCodes.Castclass, clitype) def getobj(self): raise NotImplementedError @@ -170,7 +170,7 @@ def getobj(self): return self.obj - def load(self, builder): + def load(self, methbuilder): assert False, 'XXX' ## import pdb;pdb.set_trace() ## index = self._get_index(builder) @@ -201,14 +201,14 @@ #return ootype.ooupcast(OBJECT, self.holder) return self.holder - def load(self, builder): + def load(self, methbuilder): holdertype = box(self.holder).GetType() funcfield = holdertype.GetField('func') delegatetype = self.delegatetype - index = self._get_index(builder) - self._load_from_array(builder, index, holdertype) - builder.graphbuilder.il.Emit(OpCodes.Ldfld, funcfield) - builder.graphbuilder.il.Emit(OpCodes.Castclass, delegatetype) + index = self._get_index(methbuilder) + self._load_from_array(methbuilder, index, holdertype) + methbuilder.il.Emit(OpCodes.Ldfld, funcfield) + methbuilder.il.Emit(OpCodes.Castclass, delegatetype) @specialize.arg(1) def revealconst(self, T): @@ -227,9 +227,9 @@ def getobj(self): return self.flexswitch - def load(self, builder): - index = self._get_index(builder) - self._load_from_array(builder, index, self.flexswitch.GetType()) + def load(self, methbuilder): + index = self._get_index(methbuilder) + self._load_from_array(methbuilder, index, self.flexswitch.GetType()) class Label(GenLabel): @@ -342,7 +342,7 @@ class GraphInfo: def __init__(self): self.has_flexswitches = False - self.blocks = [] # blockid -> (methodbuilder, label) + self.blocks = [] # blockid -> (methbuilder, label) class MethodBuilder(GenBuilder): @@ -506,34 +506,34 @@ class BranchBuilder(GenBuilder): - def __init__(self, graphbuilder, label): - self.graphbuilder = graphbuilder - self.rgenop = graphbuilder.rgenop + def __init__(self, methbuilder, label): + self.methbuilder = methbuilder + self.rgenop = methbuilder.rgenop self.label = label self.operations = [] self.is_open = False - self.genconsts = graphbuilder.genconsts + self.genconsts = methbuilder.genconsts def start_writing(self): self.is_open = True def finish_and_return(self, sigtoken, gv_returnvar): - op = ops.Return(self, gv_returnvar) + op = ops.Return(self.methbuilder, gv_returnvar) self.appendop(op) self.is_open = False def finish_and_goto(self, outputargs_gv, target): inputargs_gv = target.inputargs_gv assert len(inputargs_gv) == len(outputargs_gv) - op = ops.FollowLink(self, outputargs_gv, inputargs_gv, - target.label) + op = ops.FollowLink(self.methbuilder, outputargs_gv, + inputargs_gv, target.label) self.appendop(op) self.is_open = False @specialize.arg(1) def genop1(self, opname, gv_arg): opcls = ops.getopclass1(opname) - op = opcls(self, gv_arg) + op = opcls(self.methbuilder, gv_arg) self.appendop(op) gv_res = op.gv_res() return gv_res @@ -541,28 +541,28 @@ @specialize.arg(1) def genop2(self, opname, gv_arg1, gv_arg2): opcls = ops.getopclass2(opname) - op = opcls(self, gv_arg1, gv_arg2) + op = opcls(self.methbuilder, gv_arg1, gv_arg2) self.appendop(op) gv_res = op.gv_res() return gv_res def genop_call(self, sigtoken, gv_fnptr, args_gv): - op = ops.Call(self, sigtoken, gv_fnptr, args_gv) + op = ops.Call(self.methbuilder, sigtoken, gv_fnptr, args_gv) self.appendop(op) return op.gv_res() def genop_same_as(self, gv_x): - op = ops.SameAs(self, gv_x) + op = ops.SameAs(self.methbuilder, gv_x) self.appendop(op) return op.gv_res() def genop_oogetfield(self, fieldtoken, gv_obj): - op = ops.GetField(self, gv_obj, fieldtoken) + op = ops.GetField(self.methbuilder, gv_obj, fieldtoken) self.appendop(op) return op.gv_res() def genop_oosetfield(self, fieldtoken, gv_obj, gv_value): - op = ops.SetField(self, gv_obj, gv_value, fieldtoken) + op = ops.SetField(self.methbuilder, gv_obj, gv_value, fieldtoken) self.appendop(op) def enter_next_block(self, args_gv): @@ -570,28 +570,28 @@ for i in range(len(args_gv)): gv = args_gv[i] if isinstance(gv, GenConst) or gv in seen: - op = ops.SameAs(self, gv) + op = ops.SameAs(self.methbuilder, gv) self.appendop(op) args_gv[i] = op.gv_res() else: seen[gv] = None - label = self.graphbuilder.il.DefineLabel() - self.appendop(ops.MarkLabel(self, label)) + label = self.methbuilder.il.DefineLabel() + self.appendop(ops.MarkLabel(self.methbuilder, label)) return self.create_label(label, args_gv) def create_label(self, label, args_gv): - blocks = self.graphbuilder.graphinfo.blocks + blocks = self.methbuilder.graphinfo.blocks blockid = len(blocks) result = Label(blockid, label, args_gv) - blocks.append((self.graphbuilder, result)) + blocks.append((self.methbuilder, result)) return result def _jump_if(self, gv_condition, opcode): - label = self.graphbuilder.il.DefineLabel() - op = ops.Branch(self, gv_condition, opcode, label) + label = self.methbuilder.il.DefineLabel() + op = ops.Branch(self.methbuilder, gv_condition, opcode, label) self.appendop(op) - branch = BranchBuilder(self.graphbuilder, label) - self.graphbuilder.appendbranch(branch) + branch = BranchBuilder(self.methbuilder, label) + self.methbuilder.appendbranch(branch) return branch def jump_if_false(self, gv_condition, args_for_jump_gv): @@ -601,18 +601,18 @@ return self._jump_if(gv_condition, OpCodes.Brtrue) def flexswitch(self, gv_exitswitch, args_gv): - # XXX: this code is valid only for GraphBuilder - self.graphbuilder.setup_flexswitches() + # XXX: this code is valid only for Methbuilder + self.methbuilder.setup_flexswitches() flexswitch = IntFlexSwitch() - flexswitch.xxxbuilder = BranchBuilder(self.graphbuilder, self.graphbuilder.il.DefineLabel()) + flexswitch.xxxbuilder = BranchBuilder(self.methbuilder, self.methbuilder.il.DefineLabel()) gv_flexswitch = flexswitch.gv_flexswitch - lbl = self.graphbuilder.il.DefineLabel() + lbl = self.methbuilder.il.DefineLabel() default_label = self.create_label(lbl, args_gv) flexswitch.llflexswitch.set_default_blockid(default_label.blockid) - op = ops.DoFlexSwitch(self, gv_flexswitch, gv_exitswitch, args_gv) + op = ops.DoFlexSwitch(self.methbuilder, gv_flexswitch, gv_exitswitch, args_gv) self.appendop(op) - default_builder = BranchBuilder(self.graphbuilder, default_label.label) - self.graphbuilder.appendbranch(default_builder) + default_builder = BranchBuilder(self.methbuilder, default_label.label) + self.methbuilder.appendbranch(default_builder) self.is_open = False return flexswitch, default_builder @@ -620,11 +620,11 @@ self.operations.append(op) def end(self): - self.graphbuilder.end() + self.methbuilder.end() def replayops(self): assert not self.is_open - il = self.graphbuilder.il + il = self.methbuilder.il il.MarkLabel(self.label) for op in self.operations: op.emit() From antocuni at codespeak.net Wed Jul 23 10:23:22 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 23 Jul 2008 10:23:22 +0200 (CEST) Subject: [pypy-svn] r56730 - in pypy/branch/oo-jit/pypy/jit/codegen/cli: . test Message-ID: <20080723082322.16993169EEE@codespeak.net> Author: antocuni Date: Wed Jul 23 10:23:17 2008 New Revision: 56730 Added: pypy/branch/oo-jit/pypy/jit/codegen/cli/methodfactory.py - copied, changed from r56658, pypy/branch/oo-jit/pypy/jit/codegen/cli/methodbuilder.py Removed: pypy/branch/oo-jit/pypy/jit/codegen/cli/methodbuilder.py Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Log: finally fix a design that grew badly for historical reasons: - rename GraphBuilder and MethodBuilder to GraphGenerator and MethodGenerator; they are no longer GenBuilder now - the only subclass of GenBuilder is BranchBuilder; rgenop.newgraph() returns the first branch of the GraphGenerator - rename methbuilder.py to methfactory.py, and inside the module AbstractMethodBuilder to AbstractMethodWrapper etc.; there where confusion because there is a .NET class whose name is exactly MethodBuilder - rename methbuilder to meth everywhere Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Wed Jul 23 10:23:17 2008 @@ -17,7 +17,7 @@ if self._gv_res is None: restype = self.restype() if restype is not None: - loc = self.methbuilder.il.DeclareLocal(restype) + loc = self.meth.il.DeclareLocal(restype) self._gv_res = GenLocalVar(loc) return self._gv_res @@ -28,38 +28,38 @@ raise NotImplementedError def storeResult(self): - self.gv_res().store(self.methbuilder) + self.gv_res().store(self.meth) class UnaryOp(Operation): - def __init__(self, methbuilder, gv_x): - self.methbuilder = methbuilder + def __init__(self, meth, gv_x): + self.meth = meth self.gv_x = gv_x def pushAllArgs(self): - self.gv_x.load(self.methbuilder) + self.gv_x.load(self.meth) def emit(self): self.pushAllArgs() - self.methbuilder.il.Emit(self.getOpCode()) + self.meth.il.Emit(self.getOpCode()) self.storeResult() def getOpCode(self): raise NotImplementedError class BinaryOp(Operation): - def __init__(self, methbuilder, gv_x, gv_y): - self.methbuilder = methbuilder + def __init__(self, meth, gv_x, gv_y): + self.meth = meth self.gv_x = gv_x self.gv_y = gv_y def pushAllArgs(self): - self.gv_x.load(self.methbuilder) - self.gv_y.load(self.methbuilder) + self.gv_x.load(self.meth) + self.gv_y.load(self.meth) def emit(self): self.pushAllArgs() - self.methbuilder.il.Emit(self.getOpCode()) + self.meth.il.Emit(self.getOpCode()) self.storeResult() def getOpCode(self): @@ -69,25 +69,25 @@ class SameAs(UnaryOp): def emit(self): gv_res = self.gv_res() - self.gv_x.load(self.methbuilder) - self.gv_res().store(self.methbuilder) + self.gv_x.load(self.meth) + self.gv_res().store(self.meth) class MarkLabel(Operation): - def __init__(self, methbuilder, label): - self.methbuilder = methbuilder + def __init__(self, meth, label): + self.meth = meth self.label = label def restype(self): return None def emit(self): - self.methbuilder.il.MarkLabel(self.label) + self.meth.il.MarkLabel(self.label) class FollowLink(Operation): - def __init__(self, methbuilder, outputargs_gv, inputargs_gv, label): - self.methbuilder = methbuilder + def __init__(self, meth, outputargs_gv, inputargs_gv, label): + self.meth = meth self.outputargs_gv = outputargs_gv self.inputargs_gv = inputargs_gv self.label = label @@ -97,15 +97,15 @@ def emit(self): for i in range(len(self.outputargs_gv)): - self.outputargs_gv[i].load(self.methbuilder) - self.inputargs_gv[i].store(self.methbuilder) - self.methbuilder.il.Emit(OpCodes.Br, self.label) + self.outputargs_gv[i].load(self.meth) + self.inputargs_gv[i].store(self.meth) + self.meth.il.Emit(OpCodes.Br, self.label) class Branch(Operation): - def __init__(self, methbuilder, gv_cond, opcode, label): - self.methbuilder = methbuilder + def __init__(self, meth, gv_cond, opcode, label): + self.meth = meth self.gv_cond = gv_cond self.opcode = opcode self.label = label @@ -115,31 +115,31 @@ def emit(self): if self.gv_cond is not None: - self.gv_cond.load(self.methbuilder) - self.methbuilder.il.Emit(self.opcode, self.label) + self.gv_cond.load(self.meth) + self.meth.il.Emit(self.opcode, self.label) class Return(Operation): - def __init__(self, methbuilder, gv_x): - self.methbuilder = methbuilder + def __init__(self, meth, gv_x): + self.meth = meth self.gv_x = gv_x def restype(self): return None def emit(self): - retvar = self.methbuilder.retvar - retlabel = self.methbuilder.retlabel + retvar = self.meth.retvar + retlabel = self.meth.retlabel if self.gv_x is not None: - self.gv_x.load(self.methbuilder) - self.methbuilder.il.Emit(OpCodes.Stloc, retvar) - self.methbuilder.il.Emit(OpCodes.Br, retlabel) + self.gv_x.load(self.meth) + self.meth.il.Emit(OpCodes.Stloc, retvar) + self.meth.il.Emit(OpCodes.Br, retlabel) class Call(Operation): - def __init__(self, methbuilder, sigtoken, gv_fnptr, args_gv): + def __init__(self, meth, sigtoken, gv_fnptr, args_gv): from pypy.jit.codegen.cli.rgenop import class2type - self.methbuilder = methbuilder + self.meth = meth self.sigtoken = sigtoken self.gv_fnptr = gv_fnptr self.args_gv = args_gv @@ -152,18 +152,18 @@ from pypy.jit.codegen.cli.rgenop import class2type delegate_type = class2type(self.sigtoken.funcclass) meth_invoke = delegate_type.GetMethod('Invoke') - self.gv_fnptr.load(self.methbuilder) - self.methbuilder.il.Emit(OpCodes.Castclass, delegate_type) + self.gv_fnptr.load(self.meth) + self.meth.il.Emit(OpCodes.Castclass, delegate_type) for gv_arg in self.args_gv: - gv_arg.load(self.methbuilder) - self.methbuilder.il.EmitCall(OpCodes.Callvirt, meth_invoke, None) + gv_arg.load(self.meth) + self.meth.il.EmitCall(OpCodes.Callvirt, meth_invoke, None) self.storeResult() class GetField(Operation): - def __init__(self, methbuilder, gv_obj, fieldname): - self.methbuilder = methbuilder + def __init__(self, meth, gv_obj, fieldname): + self.meth = meth self.gv_obj = gv_obj clitype = gv_obj.getCliType() self.fieldinfo = clitype.GetField(fieldname) @@ -172,15 +172,15 @@ return self.fieldinfo.get_FieldType() def emit(self): - self.gv_obj.load(self.methbuilder) - self.methbuilder.il.Emit(OpCodes.Ldfld, self.fieldinfo) + self.gv_obj.load(self.meth) + self.meth.il.Emit(OpCodes.Ldfld, self.fieldinfo) self.storeResult() class SetField(Operation): - def __init__(self, methbuilder, gv_obj, gv_value, fieldname): - self.methbuilder = methbuilder + def __init__(self, meth, gv_obj, gv_value, fieldname): + self.meth = meth self.gv_obj = gv_obj self.gv_value = gv_value clitype = gv_obj.getCliType() @@ -190,14 +190,14 @@ return None def emit(self): - self.gv_obj.load(self.methbuilder) - self.gv_value.load(self.methbuilder) - self.methbuilder.il.Emit(OpCodes.Stfld, self.fieldinfo) + self.gv_obj.load(self.meth) + self.gv_value.load(self.meth) + self.meth.il.Emit(OpCodes.Stfld, self.fieldinfo) class DoFlexSwitch(Operation): - def __init__(self, methbuilder, gv_flexswitch, gv_exitswitch, args_gv): - self.methbuilder = methbuilder + def __init__(self, meth, gv_flexswitch, gv_exitswitch, args_gv): + self.meth = meth self.gv_flexswitch = gv_flexswitch self.gv_exitswitch = gv_exitswitch self.args_gv = args_gv # XXX: remove duplicates @@ -206,7 +206,7 @@ return None def emit(self): - mbuilder = self.methbuilder + mbuilder = self.meth il = mbuilder.il # get MethodInfo for LowLevelFlexSwitch.execute clitype = self.gv_flexswitch.flexswitch.GetType() @@ -228,16 +228,16 @@ class InputArgsManager: - def __init__(self, methbuilder, args_gv): - self.inputargs_var = methbuilder.inputargs_var - self.inputargs_clitype = methbuilder.inputargs_clitype + def __init__(self, meth, args_gv): + self.inputargs_var = meth.inputargs_var + self.inputargs_clitype = meth.inputargs_clitype self.args_gv = args_gv def basename_from_type(self, clitype): return clitype.get_Name() - def copy_from_args(self, methbuilder): - il = methbuilder.methbuilder.il + def copy_from_args(self, meth): + il = meth.meth.il inputargs_var = self.inputargs_var inputargs_clitype = self.inputargs_clitype counters = {} @@ -255,15 +255,15 @@ class WriteLine(Operation): - def __init__(self, methbuilder, message): - self.methbuilder = methbuilder + def __init__(self, meth, message): + self.meth = meth self.message = message def restype(self): return None def emit(self): - self.methbuilder.il.EmitWriteLine(self.message) + self.meth.il.EmitWriteLine(self.message) def opcode2attrname(opcode): if opcode == 'ldc.r8 0': @@ -327,7 +327,7 @@ if 'call' in step: return # XXX, fix this attrname = opcode2attrname(step) - body.append('self.methbuilder.il.Emit(OpCodes.%s)' % attrname) + body.append('self.meth.il.Emit(OpCodes.%s)' % attrname) elif isinstance(step, cli_opcodes.MapException): return # XXX, TODO else: Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Wed Jul 23 10:23:17 2008 @@ -7,7 +7,7 @@ from pypy.jit.codegen.model import GenVarOrConst, GenVar, GenConst from pypy.jit.codegen.model import CodeGenSwitch from pypy.jit.codegen.cli import operation as ops -from pypy.jit.codegen.cli.methodbuilder import get_methodbuilder +from pypy.jit.codegen.cli.methodfactory import get_method_wrapper from pypy.translator.cli.dotnet import CLR, typeof, new_array, init_array from pypy.translator.cli.dotnet import box, unbox, clidowncast, classof from pypy.translator.cli import dotnet @@ -60,20 +60,20 @@ def getCliType(self): return self.cliType - def load(self, methbuilder): + def load(self, meth): if self.index == 0: - methbuilder.il.Emit(OpCodes.Ldarg_0) + meth.il.Emit(OpCodes.Ldarg_0) elif self.index == 1: - methbuilder.il.Emit(OpCodes.Ldarg_1) + meth.il.Emit(OpCodes.Ldarg_1) elif self.index == 2: - methbuilder.il.Emit(OpCodes.Ldarg_2) + meth.il.Emit(OpCodes.Ldarg_2) elif self.index == 3: - methbuilder.il.Emit(OpCodes.Ldarg_3) + meth.il.Emit(OpCodes.Ldarg_3) else: - methbuilder.il.Emit(OpCodes.Ldarg, self.index) + meth.il.Emit(OpCodes.Ldarg, self.index) - def store(self, methbuilder): - methbuilder.il.Emit(OpCodes.Starg, self.index) + def store(self, meth): + meth.il.Emit(OpCodes.Starg, self.index) def __repr__(self): return "GenArgVar(%d)" % self.index @@ -85,11 +85,11 @@ def getCliType(self): return self.v.get_LocalType() - def load(self, methbuilder): - methbuilder.il.Emit(OpCodes.Ldloc, self.v) + def load(self, meth): + meth.il.Emit(OpCodes.Ldloc, self.v) - def store(self, methbuilder): - methbuilder.il.Emit(OpCodes.Stloc, self.v) + def store(self, meth): + meth.il.Emit(OpCodes.Stloc, self.v) class IntConst(GenConst): @@ -109,8 +109,8 @@ def getCliType(self): return class2type(self.cliclass) - def load(self, methbuilder): - methbuilder.il.Emit(OpCodes.Ldc_I4, self.value) + def load(self, meth): + meth.il.Emit(OpCodes.Ldc_I4, self.value) def __repr__(self): return "int const=%s" % self.value @@ -130,28 +130,28 @@ def getCliType(self): return typeof(System.Double) - def load(self, methbuilder): - methbuilder.il.Emit(OpCodes.Ldc_R8, self.value) + def load(self, meth): + meth.il.Emit(OpCodes.Ldc_R8, self.value) def __repr__(self): return "float const=%s" % self.value class BaseConst(GenConst): - def _get_index(self, methbuilder): + def _get_index(self, meth): # check whether there is already an index associated to this const try: - index = methbuilder.genconsts[self] + index = meth.genconsts[self] except KeyError: - index = len(methbuilder.genconsts) - methbuilder.genconsts[self] = index + index = len(meth.genconsts) + meth.genconsts[self] = index return index - def _load_from_array(self, methbuilder, index, clitype): - methbuilder.il.Emit(OpCodes.Ldarg_0) - methbuilder.il.Emit(OpCodes.Ldc_I4, index) - methbuilder.il.Emit(OpCodes.Ldelem_Ref) - methbuilder.il.Emit(OpCodes.Castclass, clitype) + def _load_from_array(self, meth, index, clitype): + meth.il.Emit(OpCodes.Ldarg_0) + meth.il.Emit(OpCodes.Ldc_I4, index) + meth.il.Emit(OpCodes.Ldelem_Ref) + meth.il.Emit(OpCodes.Castclass, clitype) def getobj(self): raise NotImplementedError @@ -170,7 +170,7 @@ def getobj(self): return self.obj - def load(self, methbuilder): + def load(self, meth): assert False, 'XXX' ## import pdb;pdb.set_trace() ## index = self._get_index(builder) @@ -201,14 +201,14 @@ #return ootype.ooupcast(OBJECT, self.holder) return self.holder - def load(self, methbuilder): + def load(self, meth): holdertype = box(self.holder).GetType() funcfield = holdertype.GetField('func') delegatetype = self.delegatetype - index = self._get_index(methbuilder) - self._load_from_array(methbuilder, index, holdertype) - methbuilder.il.Emit(OpCodes.Ldfld, funcfield) - methbuilder.il.Emit(OpCodes.Castclass, delegatetype) + index = self._get_index(meth) + self._load_from_array(meth, index, holdertype) + meth.il.Emit(OpCodes.Ldfld, funcfield) + meth.il.Emit(OpCodes.Castclass, delegatetype) @specialize.arg(1) def revealconst(self, T): @@ -227,9 +227,9 @@ def getobj(self): return self.flexswitch - def load(self, methbuilder): - index = self._get_index(methbuilder) - self._load_from_array(methbuilder, index, self.flexswitch.GetType()) + def load(self, meth): + index = self._get_index(meth) + self._load_from_array(meth, index, self.flexswitch.GetType()) class Label(GenLabel): @@ -335,21 +335,22 @@ args[i+1] = class2type(argsclass[i]) restype = class2type(sigtoken.res) delegatetype = class2type(sigtoken.funcclass) - builder = GraphBuilder(self, name, restype, args, delegatetype) - return builder, builder.gv_entrypoint, builder.inputargs_gv[:] + graph = GraphGenerator(self, name, restype, args, delegatetype) + builder = graph.branches[0] + return builder, graph.gv_entrypoint, graph.inputargs_gv[:] class GraphInfo: def __init__(self): self.has_flexswitches = False - self.blocks = [] # blockid -> (methbuilder, label) + self.blocks = [] # blockid -> (meth, label) -class MethodBuilder(GenBuilder): +class MethodGenerator: def __init__(self, rgenop, name, restype, args, delegatetype, graphinfo): self.rgenop = rgenop - self.meth = get_methodbuilder(name, restype, args) - self.il = self.meth.get_il_generator() + self.meth_wrapper = get_method_wrapper(name, restype, args) + self.il = self.meth_wrapper.get_il_generator() self.inputargs_gv = [] # we start from 1 because the 1st arg is an Object[] containing the genconsts for i in range(1, len(args)): @@ -368,63 +369,7 @@ def appendbranch(self, branchbuilder): self.branches.append(branchbuilder) - def start_writing(self): - self.branches[0].start_writing() - - def pause_writing(self, args_gv): - return self.branches[0].pause_writing(args_gv) - - def finish_and_return(self, sigtoken, gv_returnvar): - return self.branches[0].finish_and_return(sigtoken, - gv_returnvar) - - def finish_and_goto(self, outputargs_gv, target): - return self.branches[0].finish_and_goto(outputargs_gv, target) - - def enter_next_block(self, args_gv): - return self.branches[0].enter_next_block(args_gv) - - def jump_if_false(self, gv_condition, args_for_jump_gv): - return self.branches[0].jump_if_false(gv_condition, - args_for_jump_gv) - - def jump_if_true(self, gv_condition, args_for_jump_gv): - return self.branches[0].jump_if_true(gv_condition, - args_for_jump_gv) - - @specialize.arg(1) - def genop1(self, opname, gv_arg): - return self.branches[0].genop1(opname, gv_arg) - - @specialize.arg(1) - def genop2(self, opname, gv_arg1, gv_arg2): - return self.branches[0].genop2(opname, gv_arg1, gv_arg2) - - def genop_call(self, sigtoken, gv_fnptr, args_gv): - return self.branches[0].genop_call(sigtoken, gv_fnptr, args_gv) - - def genop_same_as(self, gv_x): - return self.branches[0].genop_same_as(gv_x) - - def genop_oogetfield(self, fieldtoken, gv_obj): - return self.branches[0].genop_oogetfield(fieldtoken, gv_obj) - - def genop_oosetfield(self, fieldtoken, gv_obj, gv_value): - return self.branches[0].genop_oosetfield(fieldtoken, gv_obj, gv_value) - - def genop_oonewarray(self, alloctoken, gv_length): - raise NotImplementedError - - def genop_oononnull(self, gv_obj): - raise NotImplementedError - - def genop_ooisnull(self, gv_obj): - raise NotImplementedError - - def flexswitch(self, gv_exitswitch, args_gv): - return self.branches[0].flexswitch(gv_exitswitch, args_gv) - - def end(self): + def emit_code(self): # emit initialization code self.emit_preamble() @@ -446,7 +391,7 @@ for gv_const, i in self.genconsts.iteritems(): consts[i] = gv_const.getobj() # build the delegate - myfunc = self.meth.create_delegate(self.delegatetype, consts) + myfunc = self.meth_wrapper.create_delegate(self.delegatetype, consts) self.gv_entrypoint.holder.SetFunc(myfunc) def emit_preamble(self): @@ -456,10 +401,10 @@ pass -class GraphBuilder(MethodBuilder): +class GraphGenerator(MethodGenerator): def __init__(self, rgenop, name, restype, args, delegatetype): graphinfo = GraphInfo() - MethodBuilder.__init__(self, rgenop, name, restype, args, delegatetype, graphinfo) + MethodGenerator.__init__(self, rgenop, name, restype, args, delegatetype, graphinfo) def setup_flexswitches(self): if self.graphinfo.has_flexswitches: @@ -502,30 +447,28 @@ self.il.Emit(OpCodes.Call, meth) - - class BranchBuilder(GenBuilder): - def __init__(self, methbuilder, label): - self.methbuilder = methbuilder - self.rgenop = methbuilder.rgenop + def __init__(self, meth, label): + self.meth = meth + self.rgenop = meth.rgenop self.label = label self.operations = [] self.is_open = False - self.genconsts = methbuilder.genconsts + self.genconsts = meth.genconsts def start_writing(self): self.is_open = True def finish_and_return(self, sigtoken, gv_returnvar): - op = ops.Return(self.methbuilder, gv_returnvar) + op = ops.Return(self.meth, gv_returnvar) self.appendop(op) self.is_open = False def finish_and_goto(self, outputargs_gv, target): inputargs_gv = target.inputargs_gv assert len(inputargs_gv) == len(outputargs_gv) - op = ops.FollowLink(self.methbuilder, outputargs_gv, + op = ops.FollowLink(self.meth, outputargs_gv, inputargs_gv, target.label) self.appendop(op) self.is_open = False @@ -533,7 +476,7 @@ @specialize.arg(1) def genop1(self, opname, gv_arg): opcls = ops.getopclass1(opname) - op = opcls(self.methbuilder, gv_arg) + op = opcls(self.meth, gv_arg) self.appendop(op) gv_res = op.gv_res() return gv_res @@ -541,28 +484,28 @@ @specialize.arg(1) def genop2(self, opname, gv_arg1, gv_arg2): opcls = ops.getopclass2(opname) - op = opcls(self.methbuilder, gv_arg1, gv_arg2) + op = opcls(self.meth, gv_arg1, gv_arg2) self.appendop(op) gv_res = op.gv_res() return gv_res def genop_call(self, sigtoken, gv_fnptr, args_gv): - op = ops.Call(self.methbuilder, sigtoken, gv_fnptr, args_gv) + op = ops.Call(self.meth, sigtoken, gv_fnptr, args_gv) self.appendop(op) return op.gv_res() def genop_same_as(self, gv_x): - op = ops.SameAs(self.methbuilder, gv_x) + op = ops.SameAs(self.meth, gv_x) self.appendop(op) return op.gv_res() def genop_oogetfield(self, fieldtoken, gv_obj): - op = ops.GetField(self.methbuilder, gv_obj, fieldtoken) + op = ops.GetField(self.meth, gv_obj, fieldtoken) self.appendop(op) return op.gv_res() def genop_oosetfield(self, fieldtoken, gv_obj, gv_value): - op = ops.SetField(self.methbuilder, gv_obj, gv_value, fieldtoken) + op = ops.SetField(self.meth, gv_obj, gv_value, fieldtoken) self.appendop(op) def enter_next_block(self, args_gv): @@ -570,28 +513,28 @@ for i in range(len(args_gv)): gv = args_gv[i] if isinstance(gv, GenConst) or gv in seen: - op = ops.SameAs(self.methbuilder, gv) + op = ops.SameAs(self.meth, gv) self.appendop(op) args_gv[i] = op.gv_res() else: seen[gv] = None - label = self.methbuilder.il.DefineLabel() - self.appendop(ops.MarkLabel(self.methbuilder, label)) + label = self.meth.il.DefineLabel() + self.appendop(ops.MarkLabel(self.meth, label)) return self.create_label(label, args_gv) def create_label(self, label, args_gv): - blocks = self.methbuilder.graphinfo.blocks + blocks = self.meth.graphinfo.blocks blockid = len(blocks) result = Label(blockid, label, args_gv) - blocks.append((self.methbuilder, result)) + blocks.append((self.meth, result)) return result def _jump_if(self, gv_condition, opcode): - label = self.methbuilder.il.DefineLabel() - op = ops.Branch(self.methbuilder, gv_condition, opcode, label) + label = self.meth.il.DefineLabel() + op = ops.Branch(self.meth, gv_condition, opcode, label) self.appendop(op) - branch = BranchBuilder(self.methbuilder, label) - self.methbuilder.appendbranch(branch) + branch = BranchBuilder(self.meth, label) + self.meth.appendbranch(branch) return branch def jump_if_false(self, gv_condition, args_for_jump_gv): @@ -601,18 +544,18 @@ return self._jump_if(gv_condition, OpCodes.Brtrue) def flexswitch(self, gv_exitswitch, args_gv): - # XXX: this code is valid only for Methbuilder - self.methbuilder.setup_flexswitches() + # XXX: this code is valid only for Meth + self.meth.setup_flexswitches() flexswitch = IntFlexSwitch() - flexswitch.xxxbuilder = BranchBuilder(self.methbuilder, self.methbuilder.il.DefineLabel()) + flexswitch.xxxbuilder = BranchBuilder(self.meth, self.meth.il.DefineLabel()) gv_flexswitch = flexswitch.gv_flexswitch - lbl = self.methbuilder.il.DefineLabel() + lbl = self.meth.il.DefineLabel() default_label = self.create_label(lbl, args_gv) flexswitch.llflexswitch.set_default_blockid(default_label.blockid) - op = ops.DoFlexSwitch(self.methbuilder, gv_flexswitch, gv_exitswitch, args_gv) + op = ops.DoFlexSwitch(self.meth, gv_flexswitch, gv_exitswitch, args_gv) self.appendop(op) - default_builder = BranchBuilder(self.methbuilder, default_label.label) - self.methbuilder.appendbranch(default_builder) + default_builder = BranchBuilder(self.meth, default_label.label) + self.meth.appendbranch(default_builder) self.is_open = False return flexswitch, default_builder @@ -620,11 +563,11 @@ self.operations.append(op) def end(self): - self.methbuilder.end() + self.meth.emit_code() def replayops(self): assert not self.is_open - il = self.methbuilder.il + il = self.meth.il il.MarkLabel(self.label) for op in self.operations: op.emit() Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Wed Jul 23 10:23:17 2008 @@ -105,25 +105,25 @@ def test_dump_assembly(self): import os - from pypy.jit.codegen.cli import methodbuilder + from pypy.jit.codegen.cli import methodfactory # clear the global state, setup env - methodbuilder.assemblyData = methodbuilder.AssemblyData() + methodfactory.assemblyData = methodfactory.AssemblyData() oldenv = os.environ.get('PYPYJITLOG') os.environ['PYPYJITLOG'] = 'generated.dll' try: self.test_adder_compile() finally: # reset the global state, clear env - methodbuilder.assemblyData = methodbuilder.AssemblyData() + methodfactory.assemblyData = methodfactory.AssemblyData() if oldenv: os.environ['PYPYJITLOG'] = oldenv else: del os.environ['PYPYJITLOG'] - f = py.path.local('generated.dll') - assert f.check() - f.remove() + f = py.path.local('generated.dll') + assert f.check() + f.remove() def test_largedummy_compile(self): py.test.skip('it works only if we increase .maxstack') From antocuni at codespeak.net Wed Jul 23 11:27:45 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 23 Jul 2008 11:27:45 +0200 (CEST) Subject: [pypy-svn] r56731 - pypy/branch/oo-jit/pypy/jit/codegen/cli Message-ID: <20080723092745.08B692A0190@codespeak.net> Author: antocuni Date: Wed Jul 23 11:27:43 2008 New Revision: 56731 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Log: sanitize the way labels and branches are handled: - use better name to distinguish labels: "label" is used for Label() instances; il_label and il_*_label are used for IL lables (i.e. those created by il.DefineLabel()) - move the logic about branch&block creation to MethodGenerator Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Wed Jul 23 11:27:43 2008 @@ -74,23 +74,23 @@ class MarkLabel(Operation): - def __init__(self, meth, label): + def __init__(self, meth, il_label): self.meth = meth - self.label = label + self.il_label = il_label def restype(self): return None def emit(self): - self.meth.il.MarkLabel(self.label) + self.meth.il.MarkLabel(self.il_label) class FollowLink(Operation): - def __init__(self, meth, outputargs_gv, inputargs_gv, label): + def __init__(self, meth, outputargs_gv, inputargs_gv, il_label): self.meth = meth self.outputargs_gv = outputargs_gv self.inputargs_gv = inputargs_gv - self.label = label + self.il_label = il_label def restype(self): return None @@ -99,16 +99,15 @@ for i in range(len(self.outputargs_gv)): self.outputargs_gv[i].load(self.meth) self.inputargs_gv[i].store(self.meth) - self.meth.il.Emit(OpCodes.Br, self.label) - + self.meth.il.Emit(OpCodes.Br, self.il_label) class Branch(Operation): - def __init__(self, meth, gv_cond, opcode, label): + def __init__(self, meth, gv_cond, opcode, il_label): self.meth = meth self.gv_cond = gv_cond self.opcode = opcode - self.label = label + self.il_label = il_label def restype(self): return None @@ -116,7 +115,7 @@ def emit(self): if self.gv_cond is not None: self.gv_cond.load(self.meth) - self.meth.il.Emit(self.opcode, self.label) + self.meth.il.Emit(self.opcode, self.il_label) class Return(Operation): @@ -129,11 +128,11 @@ def emit(self): retvar = self.meth.retvar - retlabel = self.meth.retlabel + il_retlabel = self.meth.il_retlabel if self.gv_x is not None: self.gv_x.load(self.meth) self.meth.il.Emit(OpCodes.Stloc, retvar) - self.meth.il.Emit(OpCodes.Br, retlabel) + self.meth.il.Emit(OpCodes.Br, il_retlabel) class Call(Operation): @@ -223,7 +222,7 @@ il.Emit(OpCodes.Ldloc, mbuilder.inputargs_var) il.Emit(OpCodes.Callvirt, meth_execute) il.Emit(OpCodes.Stloc, mbuilder.jumpto_var) - il.Emit(OpCodes.Br, mbuilder.dispatch_jump_label) + il.Emit(OpCodes.Br, mbuilder.il_dispatch_jump_label) class InputArgsManager: Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Wed Jul 23 11:27:43 2008 @@ -233,9 +233,9 @@ class Label(GenLabel): - def __init__(self, blockid, label, inputargs_gv): + def __init__(self, blockid, il_label, inputargs_gv): self.blockid = blockid - self.label = label + self.il_label = il_label self.inputargs_gv = inputargs_gv @@ -358,16 +358,26 @@ self.delegatetype = delegatetype self.gv_entrypoint = FunctionConst(delegatetype) self.genconsts = {} - self.branches = [BranchBuilder(self, self.il.DefineLabel())] + self.branches = [] + self.newbranch() if restype is not None: self.retvar = self.il.DeclareLocal(restype) else: self.retvar = None - self.retlabel = self.il.DefineLabel() + self.il_retlabel = self.il.DefineLabel() self.graphinfo = graphinfo - def appendbranch(self, branchbuilder): - self.branches.append(branchbuilder) + def newbranch(self): + branch = BranchBuilder(self, self.il.DefineLabel()) + self.branches.append(branch) + return branch + + def newblock(self, args_gv): + blocks = self.graphinfo.blocks + blockid = len(blocks) + result = Label(blockid, self.il.DefineLabel(), args_gv) + blocks.append((self, result)) + return result def emit_code(self): # emit initialization code @@ -381,7 +391,7 @@ self.emit_before_returnblock() # render the return block for last, else the verifier could complain - self.il.MarkLabel(self.retlabel) + self.il.MarkLabel(self.il_retlabel) if self.retvar: self.il.Emit(OpCodes.Ldloc, self.retvar) self.il.Emit(OpCodes.Ret) @@ -410,7 +420,7 @@ if self.graphinfo.has_flexswitches: return self.graphinfo.has_flexswitches = True - self.dispatch_jump_label = self.il.DefineLabel() + self.il_dispatch_jump_label = self.il.DefineLabel() self.inputargs_clitype = class2type(cInputArgs) self.inputargs_var = self.il.DeclareLocal(self.inputargs_clitype) self.jumpto_var = self.il.DeclareLocal(class2type(cInt32)) @@ -428,17 +438,17 @@ if not self.graphinfo.has_flexswitches: return # make sure we don't enter dispatch_jump by mistake - self.il.Emit(OpCodes.Br, self.retlabel) - self.il.MarkLabel(self.dispatch_jump_label) + self.il.Emit(OpCodes.Br, self.il_retlabel) + self.il.MarkLabel(self.il_dispatch_jump_label) - labels = new_array(System.Reflection.Emit.Label, + il_labels = new_array(System.Reflection.Emit.Label, len(self.graphinfo.blocks)) - for blockid, (builder, genlabel) in self.graphinfo.blocks: + for blockid, (builder, label) in self.graphinfo.blocks: assert builder is self - labels[blockid] = genlabel.label + il_labels[blockid] = label.il_label self.il.Emit(OpCodes.Ldloc, self.jumpto_var) - self.il.Emit(OpCodes.Switch, labels) + self.il.Emit(OpCodes.Switch, il_labels) # XXX: handle blockids that are inside flexswitch cases # default: Utils.throwInvalidBlockId(jumpto) clitype = class2type(cUtils) @@ -449,10 +459,10 @@ class BranchBuilder(GenBuilder): - def __init__(self, meth, label): + def __init__(self, meth, il_label): self.meth = meth self.rgenop = meth.rgenop - self.label = label + self.il_label = il_label self.operations = [] self.is_open = False self.genconsts = meth.genconsts @@ -465,11 +475,11 @@ self.appendop(op) self.is_open = False - def finish_and_goto(self, outputargs_gv, target): - inputargs_gv = target.inputargs_gv + def finish_and_goto(self, outputargs_gv, label): + inputargs_gv = label.inputargs_gv assert len(inputargs_gv) == len(outputargs_gv) op = ops.FollowLink(self.meth, outputargs_gv, - inputargs_gv, target.label) + inputargs_gv, label.il_label) self.appendop(op) self.is_open = False @@ -518,23 +528,14 @@ args_gv[i] = op.gv_res() else: seen[gv] = None - label = self.meth.il.DefineLabel() - self.appendop(ops.MarkLabel(self.meth, label)) - return self.create_label(label, args_gv) - - def create_label(self, label, args_gv): - blocks = self.meth.graphinfo.blocks - blockid = len(blocks) - result = Label(blockid, label, args_gv) - blocks.append((self.meth, result)) - return result + label = self.meth.newblock(args_gv) + self.appendop(ops.MarkLabel(self.meth, label.il_label)) + return label def _jump_if(self, gv_condition, opcode): - label = self.meth.il.DefineLabel() - op = ops.Branch(self.meth, gv_condition, opcode, label) + branch = self.meth.newbranch() + op = ops.Branch(self.meth, gv_condition, opcode, branch.il_label) self.appendop(op) - branch = BranchBuilder(self.meth, label) - self.meth.appendbranch(branch) return branch def jump_if_false(self, gv_condition, args_for_jump_gv): @@ -544,20 +545,19 @@ return self._jump_if(gv_condition, OpCodes.Brtrue) def flexswitch(self, gv_exitswitch, args_gv): - # XXX: this code is valid only for Meth + # XXX: this code is valid only for GraphGenerator self.meth.setup_flexswitches() flexswitch = IntFlexSwitch() - flexswitch.xxxbuilder = BranchBuilder(self.meth, self.meth.il.DefineLabel()) + flexswitch.xxxbuilder = self.meth.newbranch() gv_flexswitch = flexswitch.gv_flexswitch - lbl = self.meth.il.DefineLabel() - default_label = self.create_label(lbl, args_gv) - flexswitch.llflexswitch.set_default_blockid(default_label.blockid) - op = ops.DoFlexSwitch(self.meth, gv_flexswitch, gv_exitswitch, args_gv) + default_branch = self.meth.newbranch() + label = default_branch.label + flexswitch.llflexswitch.set_default_blockid(label.blockid) + op = ops.DoFlexSwitch(self.meth, gv_flexswitch, + gv_exitswitch, args_gv) self.appendop(op) - default_builder = BranchBuilder(self.meth, default_label.label) - self.meth.appendbranch(default_builder) self.is_open = False - return flexswitch, default_builder + return flexswitch, default_branch def appendop(self, op): self.operations.append(op) @@ -568,7 +568,7 @@ def replayops(self): assert not self.is_open il = self.meth.il - il.MarkLabel(self.label) + il.MarkLabel(self.il_label) for op in self.operations: op.emit() From hpk at codespeak.net Wed Jul 23 17:24:11 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Wed, 23 Jul 2008 17:24:11 +0200 (CEST) Subject: [pypy-svn] r56740 - pypy/extradoc/talk/pycon-uk-2008 Message-ID: <20080723152411.CAB07169F75@codespeak.net> Author: hpk Date: Wed Jul 23 17:24:09 2008 New Revision: 56740 Added: pypy/extradoc/talk/pycon-uk-2008/pypy-python-status.txt Log: status/plan talk draft, mainly about pypy's python. Added: pypy/extradoc/talk/pycon-uk-2008/pypy-python-status.txt ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pycon-uk-2008/pypy-python-status.txt Wed Jul 23 17:24:09 2008 @@ -0,0 +1,27 @@ +PyPy's Python status and plans +==================================== + +In this talk we'd like to discuss the status and plans +for the near and mid-term future of PyPy's Python Interpreter, +particularly its upcoming 1.1 release. We'll look at what +kind of applications you can already run and where more +work and help is needed. The talk also aims to clarify what +is the "PyPy technology cloud". There also should be room to +discuss and demo some current highlights such as PyPy's +robust virtualization/sandboxing approach for Python programs. + +level: beginner to intermediate + +biography +------------ + +Holger Krekel is a co-founder of the PyPy project and +participates on many levels in it. He is the initiator and +maintainer of the popular py.test and a few other tools. In +2004 he founded merlinux, a company organising itself +virtually, mostly with european developers from PyPy and +py.test contexts. + +Contact: holger at merlinux.eu + + From hpk at codespeak.net Wed Jul 23 17:24:19 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Wed, 23 Jul 2008 17:24:19 +0200 (CEST) Subject: [pypy-svn] r56741 - pypy/extradoc/talk/pycon-uk-2008 Message-ID: <20080723152419.89A15169F85@codespeak.net> Author: hpk Date: Wed Jul 23 17:24:18 2008 New Revision: 56741 Modified: pypy/extradoc/talk/pycon-uk-2008/pypy-jit.txt Log: fix typo. Modified: pypy/extradoc/talk/pycon-uk-2008/pypy-jit.txt ============================================================================== --- pypy/extradoc/talk/pycon-uk-2008/pypy-jit.txt (original) +++ pypy/extradoc/talk/pycon-uk-2008/pypy-jit.txt Wed Jul 23 17:24:18 2008 @@ -12,7 +12,7 @@ PyPy is a framework to implement dynamic languages that embraces this challenge. It allows us to write such languages in the form of simple interpreters, expressed in a rich subset of Python. This means that -the resulting implementations can can be evolved with reasonable +the resulting implementations can be evolved with reasonable effort. PyPy then generates a variety of virtual machines (VMs) from a single From hpk at codespeak.net Wed Jul 23 18:11:17 2008 From: hpk at codespeak.net (hpk at codespeak.net) Date: Wed, 23 Jul 2008 18:11:17 +0200 (CEST) Subject: [pypy-svn] r56745 - pypy/extradoc/talk/pycon-uk-2008 Message-ID: <20080723161117.CB8F8169F63@codespeak.net> Author: hpk Date: Wed Jul 23 18:11:15 2008 New Revision: 56745 Modified: pypy/extradoc/talk/pycon-uk-2008/pypy-python-status.txt Log: add a note about duration Modified: pypy/extradoc/talk/pycon-uk-2008/pypy-python-status.txt ============================================================================== --- pypy/extradoc/talk/pycon-uk-2008/pypy-python-status.txt (original) +++ pypy/extradoc/talk/pycon-uk-2008/pypy-python-status.txt Wed Jul 23 18:11:15 2008 @@ -11,6 +11,7 @@ robust virtualization/sandboxing approach for Python programs. level: beginner to intermediate +duration: 45 minutes biography ------------ From antocuni at codespeak.net Thu Jul 24 11:16:24 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 24 Jul 2008 11:16:24 +0200 (CEST) Subject: [pypy-svn] r56763 - in pypy/branch/oo-jit/pypy/jit/codegen/cli: . test Message-ID: <20080724091624.5DB58168001@codespeak.net> Author: antocuni Date: Thu Jul 24 11:16:23 2008 New Revision: 56763 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/methodfactory.py pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Log: yahi! The first flexswitch test passes :-) Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/methodfactory.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/methodfactory.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/methodfactory.py Thu Jul 24 11:16:23 2008 @@ -32,6 +32,7 @@ class AssemblyData: assembly = None name = None + methcount = 0 def is_enabled(self): if self.name is None: @@ -56,6 +57,8 @@ def __init__(self, name, res, args): module = assemblyData.module + name = '%s_%d' % (name, assemblyData.methcount) + assemblyData.methcount += 1 self.typeBuilder = AutoSaveAssembly.DefineType(module, name) self.meth = AutoSaveAssembly.DefineMethod(self.typeBuilder, "invoke", res, args) Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Thu Jul 24 11:16:23 2008 @@ -13,12 +13,10 @@ return self.gv_x.getCliType() def gv_res(self): - from pypy.jit.codegen.cli.rgenop import GenLocalVar if self._gv_res is None: restype = self.restype() if restype is not None: - loc = self.meth.il.DeclareLocal(restype) - self._gv_res = GenLocalVar(loc) + self._gv_res = self.meth.newlocalvar(restype) return self._gv_res def emit(self): @@ -127,12 +125,29 @@ return None def emit(self): - retvar = self.meth.retvar - il_retlabel = self.meth.il_retlabel + gv_retvar = self.meth.gv_retvar + retlabel = self.meth.retlabel if self.gv_x is not None: self.gv_x.load(self.meth) - self.meth.il.Emit(OpCodes.Stloc, retvar) - self.meth.il.Emit(OpCodes.Br, il_retlabel) + gv_retvar.store(self.meth) + self.meth.il.Emit(OpCodes.Br, retlabel.il_label) + +class ReturnFromFlexSwitch(Operation): + + def __init__(self, meth, gv_x): + self.meth = meth + self.gv_x = gv_x + + def restype(self): + return None + + def emit(self): + il = self.meth.il + manager = InputArgsManager(self.meth, [self.gv_x]) + manager.copy_from_args() + blockid = self.meth.graphinfo.graph_retlabel.blockid + il.Emit(OpCodes.Ldc_I4, blockid) + il.Emit(OpCodes.Ret) class Call(Operation): @@ -193,6 +208,10 @@ self.gv_value.load(self.meth) self.meth.il.Emit(OpCodes.Stfld, self.fieldinfo) +def mark(il, s): + il.Emit(OpCodes.Ldstr, s) + il.Emit(OpCodes.Pop) + class DoFlexSwitch(Operation): def __init__(self, meth, gv_flexswitch, gv_exitswitch, args_gv): @@ -205,40 +224,39 @@ return None def emit(self): - mbuilder = self.meth - il = mbuilder.il + graph = self.meth + il = graph.il # get MethodInfo for LowLevelFlexSwitch.execute clitype = self.gv_flexswitch.flexswitch.GetType() meth_execute = clitype.GetMethod('execute') # setup the correct inputargs - manager = InputArgsManager(mbuilder, self.args_gv) - manager.copy_from_args(mbuilder) + manager = InputArgsManager(graph, self.args_gv) + manager.copy_from_args() # jumpto = flexswitch.execute(exitswitch, inputargs); # goto dispatch_jump; - self.gv_flexswitch.load(mbuilder) - self.gv_exitswitch.load(mbuilder) - il.Emit(OpCodes.Ldloc, mbuilder.inputargs_var) + self.gv_flexswitch.load(graph) + self.gv_exitswitch.load(graph) + graph.gv_inputargs.load(graph) il.Emit(OpCodes.Callvirt, meth_execute) - il.Emit(OpCodes.Stloc, mbuilder.jumpto_var) - il.Emit(OpCodes.Br, mbuilder.il_dispatch_jump_label) + il.Emit(OpCodes.Stloc, graph.jumpto_var) + il.Emit(OpCodes.Br, graph.il_dispatch_jump_label) class InputArgsManager: def __init__(self, meth, args_gv): - self.inputargs_var = meth.inputargs_var - self.inputargs_clitype = meth.inputargs_clitype + self.meth = meth self.args_gv = args_gv def basename_from_type(self, clitype): return clitype.get_Name() - def copy_from_args(self, meth): - il = meth.meth.il - inputargs_var = self.inputargs_var - inputargs_clitype = self.inputargs_clitype + def _get_fields(self): + fields = [] + gv_inputargs = self.meth.gv_inputargs + inputargs_clitype = gv_inputargs.getCliType() counters = {} for gv_arg in self.args_gv: clitype = gv_arg.getCliType() @@ -247,11 +265,33 @@ fieldname = '%s_%d' % (basename, count) counters[clitype] = count+1 field = inputargs_clitype.GetField(fieldname) + fields.append(field) + return fields - il.Emit(OpCodes.Ldloc, inputargs_var) - gv_arg.load(builder) + def copy_from_args(self): + il = self.meth.il + gv_inputargs = self.meth.gv_inputargs + fields = self._get_fields() + assert len(self.args_gv) == len(fields) + for i in range(len(self.args_gv)): + gv_arg = self.args_gv[i] + field = fields[i] + gv_inputargs.load(self.meth) + gv_arg.load(self.meth) il.Emit(OpCodes.Stfld, field) + def copy_to_args(self): + il = self.meth.il + gv_inputargs = self.meth.gv_inputargs + fields = self._get_fields() + assert len(self.args_gv) == len(fields) + for i in range(len(self.args_gv)): + gv_arg = self.args_gv[i] + field = fields[i] + gv_inputargs.load(self.meth) + il.Emit(OpCodes.Ldfld, field) + gv_arg.store(self.meth) + class WriteLine(Operation): def __init__(self, meth, message): Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Thu Jul 24 11:16:23 2008 @@ -26,6 +26,7 @@ cChar = classof(System.Char) cInputArgs = classof(InputArgs) cUtils = classof(CLR.pypy.runtime.Utils) +cFlexSwitchCase = classof(CLR.pypy.runtime.FlexSwitchCase) class SigToken: def __init__(self, args, res, funcclass): @@ -233,11 +234,20 @@ class Label(GenLabel): - def __init__(self, blockid, il_label, inputargs_gv): + def __init__(self, blockid, il, inputargs_gv): self.blockid = blockid - self.il_label = il_label + self.il_label = il.DefineLabel() + self.il_trampoline_label = il.DefineLabel() self.inputargs_gv = inputargs_gv + def emit_trampoline(self, meth): + from pypy.jit.codegen.cli.operation import InputArgsManager + from pypy.jit.codegen.cli.operation import mark + il = meth.il + manager = InputArgsManager(meth, self.inputargs_gv) + il.MarkLabel(self.il_trampoline_label) + manager.copy_to_args() + il.Emit(OpCodes.Br, self.il_label) class RCliGenOp(AbstractRGenOp): @@ -328,14 +338,11 @@ pass def newgraph(self, sigtoken, name): - argsclass = sigtoken.args - args = new_array(System.Type, len(argsclass)+1) - args[0] = System.Type.GetType("System.Object[]") - for i in range(len(argsclass)): - args[i+1] = class2type(argsclass[i]) + arglist = [class2type(cls) for cls in sigtoken.args] restype = class2type(sigtoken.res) delegatetype = class2type(sigtoken.funcclass) - graph = GraphGenerator(self, name, restype, args, delegatetype) + graph = GraphGenerator(self, name, restype, arglist, + delegatetype) builder = graph.branches[0] return builder, graph.gv_entrypoint, graph.inputargs_gv[:] @@ -344,11 +351,16 @@ def __init__(self): self.has_flexswitches = False self.blocks = [] # blockid -> (meth, label) + self.flexswitch_meths = [] + self.main_retlabel = None class MethodGenerator: - def __init__(self, rgenop, name, restype, args, delegatetype, graphinfo): + def __init__(self, rgenop, name, restype, arglist, + delegatetype, graphinfo): self.rgenop = rgenop + self.name = name + args = self._get_args_array(arglist) self.meth_wrapper = get_method_wrapper(name, restype, args) self.il = self.meth_wrapper.get_il_generator() self.inputargs_gv = [] @@ -356,16 +368,25 @@ for i in range(1, len(args)): self.inputargs_gv.append(GenArgVar(i, args[i])) self.delegatetype = delegatetype + self.graphinfo = graphinfo + self.gv_entrypoint = FunctionConst(delegatetype) self.genconsts = {} self.branches = [] self.newbranch() - if restype is not None: - self.retvar = self.il.DeclareLocal(restype) + if restype is None: + self.gv_retvar = None + self.retlabel = self.newblock([]) else: - self.retvar = None - self.il_retlabel = self.il.DefineLabel() - self.graphinfo = graphinfo + self.gv_retvar = self.newlocalvar(restype) + self.retlabel = self.newblock([self.gv_retvar]) + + def _get_args_array(self, arglist): + array = new_array(System.Type, len(arglist)+1) + array[0] = System.Type.GetType("System.Object[]") + for i in range(len(arglist)): + array[i+1] = arglist[i] + return array def newbranch(self): branch = BranchBuilder(self, self.il.DefineLabel()) @@ -375,10 +396,19 @@ def newblock(self, args_gv): blocks = self.graphinfo.blocks blockid = len(blocks) - result = Label(blockid, self.il.DefineLabel(), args_gv) + result = Label(blockid, self.il, args_gv) blocks.append((self, result)) return result + def newlocalvar(self, clitype): + return GenLocalVar(self.il.DeclareLocal(clitype)) + + def map_genvar(self, gv_var): + return gv_var + + def get_op_Return(self, gv_returnvar): + raise NotImplementedError + def emit_code(self): # emit initialization code self.emit_preamble() @@ -390,10 +420,11 @@ # emit dispatch_jump, if there are flexswitches self.emit_before_returnblock() - # render the return block for last, else the verifier could complain - self.il.MarkLabel(self.il_retlabel) - if self.retvar: - self.il.Emit(OpCodes.Ldloc, self.retvar) + # emit the return block at last, else the verifier complains + self.il.MarkLabel(self.retlabel.il_label) + if self.gv_retvar: + self.gv_retvar.load(self) + self.il.Emit(OpCodes.Ret) # initialize the array of genconsts @@ -415,37 +446,50 @@ def __init__(self, rgenop, name, restype, args, delegatetype): graphinfo = GraphInfo() MethodGenerator.__init__(self, rgenop, name, restype, args, delegatetype, graphinfo) + graphinfo.graph_retlabel = self.retlabel def setup_flexswitches(self): if self.graphinfo.has_flexswitches: return self.graphinfo.has_flexswitches = True self.il_dispatch_jump_label = self.il.DefineLabel() - self.inputargs_clitype = class2type(cInputArgs) - self.inputargs_var = self.il.DeclareLocal(self.inputargs_clitype) + self.gv_inputargs = self.newlocalvar(class2type(cInputArgs)) self.jumpto_var = self.il.DeclareLocal(class2type(cInt32)) + def get_op_Return(self, gv_returnvar): + return ops.Return(self, gv_returnvar) + + def emit_code(self): + self.emit_flexswitches() + MethodGenerator.emit_code(self) + + def emit_flexswitches(self): + for meth in self.graphinfo.flexswitch_meths: + meth.emit_code() + def emit_preamble(self): if not self.graphinfo.has_flexswitches: return # InputArgs inputargs_var = new InputArgs() - clitype = class2type(cInputArgs) + clitype = self.gv_inputargs.getCliType() ctor = clitype.GetConstructor(new_array(System.Type, 0)) self.il.Emit(OpCodes.Newobj, ctor) - self.il.Emit(OpCodes.Stloc, self.inputargs_var) + self.gv_inputargs.store(self) def emit_before_returnblock(self): if not self.graphinfo.has_flexswitches: return # make sure we don't enter dispatch_jump by mistake - self.il.Emit(OpCodes.Br, self.il_retlabel) + self.il.Emit(OpCodes.Br, self.retlabel.il_label) self.il.MarkLabel(self.il_dispatch_jump_label) - il_labels = new_array(System.Reflection.Emit.Label, - len(self.graphinfo.blocks)) - for blockid, (builder, label) in self.graphinfo.blocks: - assert builder is self - il_labels[blockid] = label.il_label + blocks = self.graphinfo.blocks + il_labels = new_array(System.Reflection.Emit.Label, len(blocks)) + for blockid in range(len(blocks)): + builder, label = blocks[blockid] + if builder is not self: + continue # XXX FIXME + il_labels[blockid] = label.il_trampoline_label self.il.Emit(OpCodes.Ldloc, self.jumpto_var) self.il.Emit(OpCodes.Switch, il_labels) @@ -456,6 +500,51 @@ self.il.Emit(OpCodes.Ldloc, self.jumpto_var) self.il.Emit(OpCodes.Call, meth) + # emit all the trampolines to the blocks + for builder, label in blocks: + if builder is not self: + continue #XXX? + label.emit_trampoline(self) + +class FlexSwitchCaseGenerator(MethodGenerator): + flexswitch = None + value = -1 + linkargs_gv = None + linkargs_gv_map = None + + def set_parent_flexswitch(self, flexswitch, value): + self.parent_flexswitch = flexswitch + self.value = value + + def set_linkargs_gv(self, linkargs_gv): + self.linkargs_gv = linkargs_gv + self.linkargs_gv_map = {} + for gv_arg in linkargs_gv: + gv_local = self.newlocalvar(gv_arg.getCliType()) + self.linkargs_gv_map[gv_arg] = gv_local + + self.gv_inputargs = self.inputargs_gv[1] # InputArgs is always the 2nd argument + + def map_genvar(self, gv_var): + return self.linkargs_gv_map.get(gv_var, gv_var) + + def get_op_Return(self, gv_returnvar): + return ops.ReturnFromFlexSwitch(self, gv_returnvar) + + def emit_code(self): + MethodGenerator.emit_code(self) + func = self.gv_entrypoint.holder.GetFunc() + self.parent_flexswitch.llflexswitch.add_case(self.value, func) + + def emit_preamble(self): + from pypy.jit.codegen.cli.operation import InputArgsManager + linkargs_out_gv = [] + for gv_linkarg in self.linkargs_gv: + gv_var = self.linkargs_gv_map[gv_linkarg] + linkargs_out_gv.append(gv_var) + manager = InputArgsManager(self, linkargs_out_gv) + manager.copy_to_args() + class BranchBuilder(GenBuilder): @@ -471,7 +560,7 @@ self.is_open = True def finish_and_return(self, sigtoken, gv_returnvar): - op = ops.Return(self.meth, gv_returnvar) + op = self.meth.get_op_Return(gv_returnvar) self.appendop(op) self.is_open = False @@ -493,6 +582,10 @@ @specialize.arg(1) def genop2(self, opname, gv_arg1, gv_arg2): + # XXX: also other ops + gv_arg1 = self.meth.map_genvar(gv_arg1) + gv_arg2 = self.meth.map_genvar(gv_arg2) + opcls = ops.getopclass2(opname) op = opcls(self.meth, gv_arg1, gv_arg2) self.appendop(op) @@ -547,11 +640,10 @@ def flexswitch(self, gv_exitswitch, args_gv): # XXX: this code is valid only for GraphGenerator self.meth.setup_flexswitches() - flexswitch = IntFlexSwitch() - flexswitch.xxxbuilder = self.meth.newbranch() + flexswitch = IntFlexSwitch(self.meth, args_gv) gv_flexswitch = flexswitch.gv_flexswitch default_branch = self.meth.newbranch() - label = default_branch.label + label = default_branch.enter_next_block(args_gv) flexswitch.llflexswitch.set_default_blockid(label.blockid) op = ops.DoFlexSwitch(self.meth, gv_flexswitch, gv_exitswitch, args_gv) @@ -575,14 +667,26 @@ class IntFlexSwitch(CodeGenSwitch): - def __init__(self): + def __init__(self, graph, linkargs_gv): + self.graph = graph + self.linkargs_gv = linkargs_gv self.llflexswitch = LowLevelFlexSwitch() self.gv_flexswitch = FlexSwitchConst(self.llflexswitch) def add_case(self, gv_case): - return self.xxxbuilder - #import pdb;pdb.set_trace() - + graph = self.graph + name = graph.name + '_case' + restype = class2type(cInt32) + arglist = [class2type(cInt32), class2type(cInputArgs)] + delegatetype = class2type(cFlexSwitchCase) + graphinfo = graph.graphinfo + meth = FlexSwitchCaseGenerator(graph.rgenop, name, restype, + arglist, delegatetype, graphinfo) + graphinfo.flexswitch_meths.append(meth) + value = gv_case.revealconst(ootype.Signed) + meth.set_parent_flexswitch(self, value) + meth.set_linkargs_gv(self.linkargs_gv) + return meth.branches[0] global_rgenop = RCliGenOp() Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Thu Jul 24 11:16:23 2008 @@ -33,9 +33,6 @@ def test_float_loop_direct(self): py.test.skip('fixme') - def test_switch_direct(self): - py.test.skip('in-progress') - def test_large_switch_direct(self): py.test.skip('no promotion/flexswitch for now please :-)') From antocuni at codespeak.net Thu Jul 24 11:36:01 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 24 Jul 2008 11:36:01 +0200 (CEST) Subject: [pypy-svn] r56764 - in pypy/branch/oo-jit/pypy/translator/cli: . test Message-ID: <20080724093601.412F7169DFA@codespeak.net> Author: antocuni Date: Thu Jul 24 11:35:59 2008 New Revision: 56764 Modified: pypy/branch/oo-jit/pypy/translator/cli/dotnet.py pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Log: add a new operation cliupcast Modified: pypy/branch/oo-jit/pypy/translator/cli/dotnet.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/dotnet.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/dotnet.py Thu Jul 24 11:35:59 2008 @@ -648,6 +648,27 @@ return hop.genop('oodowncast', [v_inst], resulttype = hop.r_result.lowleveltype) +def cliupcast(obj, TYPE): + return obj + +class Entry(ExtRegistryEntry): + _about_ = cliupcast + + def compute_result_annotation(self, s_value, s_type): + if isinstance(s_type.const, ootype.OOType): + TYPE = s_type.const + else: + cliClass = s_type.const + TYPE = cliClass._INSTANCE + assert ootype.isSubclass(s_value.ootype, TYPE) + return SomeOOInstance(TYPE) + + def specialize_call(self, hop): + assert isinstance(hop.args_s[0], annmodel.SomeOOInstance) + v_inst = hop.inputarg(hop.args_r[0], arg=0) + return hop.genop('ooupcast', [v_inst], resulttype = hop.r_result.lowleveltype) + + def cast_to_native_object(obj): raise TypeError, "cast_to_native_object is meant to be rtyped and not called direclty" Modified: pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py (original) +++ pypy/branch/oo-jit/pypy/translator/cli/test/test_dotnet.py Thu Jul 24 11:35:59 2008 @@ -8,7 +8,7 @@ from pypy.translator.cli.dotnet import SomeCliClass, SomeCliStaticMethod,\ NativeInstance, CLR, box, unbox, OverloadingResolver, NativeException,\ native_exc, new_array, init_array, typeof, eventhandler, clidowncast,\ - classof, cast_to_native_object, cast_from_native_object + cliupcast, classof, cast_to_native_object, cast_from_native_object System = CLR.System ArrayList = CLR.System.Collections.ArrayList @@ -407,6 +407,19 @@ res = self.interpret(fn, []) assert res is None + def test_cliupcast(self): + def fn(flag): + a = ArrayList() + a.Add(None) + if flag: + obj = cliupcast(a, System.Object) + else: + obj = box(42) + b = clidowncast(obj, ArrayList) + return b.get_Item(0) + res = self.interpret(fn, [True]) + assert res is None + def test_mix_None_and_instance(self): def g(x): return x From antocuni at codespeak.net Thu Jul 24 11:39:56 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 24 Jul 2008 11:39:56 +0200 (CEST) Subject: [pypy-svn] r56765 - in pypy/branch/oo-jit/pypy/jit/codegen/cli: . test Message-ID: <20080724093956.25B30169E8F@codespeak.net> Author: antocuni Date: Thu Jul 24 11:39:55 2008 New Revision: 56765 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Log: some rpython fixes, and test_switch_compile also passes Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Thu Jul 24 11:39:55 2008 @@ -227,7 +227,7 @@ graph = self.meth il = graph.il # get MethodInfo for LowLevelFlexSwitch.execute - clitype = self.gv_flexswitch.flexswitch.GetType() + clitype = self.gv_flexswitch.llflexswitch.GetType() meth_execute = clitype.GetMethod('execute') # setup the correct inputargs Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Thu Jul 24 11:39:55 2008 @@ -14,6 +14,7 @@ System = CLR.System DelegateHolder = CLR.pypy.runtime.DelegateHolder LowLevelFlexSwitch = CLR.pypy.runtime.LowLevelFlexSwitch +FlexSwitchCase = CLR.pypy.runtime.FlexSwitchCase InputArgs = CLR.pypy.runtime.InputArgs OpCodes = System.Reflection.Emit.OpCodes @@ -26,7 +27,7 @@ cChar = classof(System.Char) cInputArgs = classof(InputArgs) cUtils = classof(CLR.pypy.runtime.Utils) -cFlexSwitchCase = classof(CLR.pypy.runtime.FlexSwitchCase) +cFlexSwitchCase = classof(FlexSwitchCase) class SigToken: def __init__(self, args, res, funcclass): @@ -198,9 +199,7 @@ self.delegatetype = delegatetype def getobj(self): - # XXX: should the conversion be done automatically? - #return ootype.ooupcast(OBJECT, self.holder) - return self.holder + return dotnet.cliupcast(self.holder, System.Object) def load(self, meth): holdertype = box(self.holder).GetType() @@ -222,15 +221,15 @@ class FlexSwitchConst(BaseConst): - def __init__(self, flexswitch): - self.flexswitch = flexswitch + def __init__(self, llflexswitch): + self.llflexswitch = llflexswitch def getobj(self): - return self.flexswitch + return dotnet.cliupcast(self.llflexswitch, System.Object) def load(self, meth): index = self._get_index(meth) - self._load_from_array(meth, index, self.flexswitch.GetType()) + self._load_from_array(meth, index, self.llflexswitch.GetType()) class Label(GenLabel): @@ -534,7 +533,8 @@ def emit_code(self): MethodGenerator.emit_code(self) func = self.gv_entrypoint.holder.GetFunc() - self.parent_flexswitch.llflexswitch.add_case(self.value, func) + func2 = clidowncast(func, FlexSwitchCase) + self.parent_flexswitch.llflexswitch.add_case(self.value, func2) def emit_preamble(self): from pypy.jit.codegen.cli.operation import InputArgsManager Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Thu Jul 24 11:39:55 2008 @@ -125,9 +125,6 @@ def test_largedummy_compile(self): py.test.skip('it works only if we increase .maxstack') - def test_switch_compile(self): - py.test.skip('no promotion/flexswitch for now please :-)') - def test_large_switch_compile(self): py.test.skip('no promotion/flexswitch for now please :-)') From antocuni at codespeak.net Thu Jul 24 14:39:48 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 24 Jul 2008 14:39:48 +0200 (CEST) Subject: [pypy-svn] r56771 - in pypy/branch/oo-jit/pypy: jit/codegen/cli/test translator/cli/src Message-ID: <20080724123948.1C088169EF7@codespeak.net> Author: antocuni Date: Thu Jul 24 14:39:47 2008 New Revision: 56771 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Log: make flexswitches growable Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Thu Jul 24 14:39:47 2008 @@ -33,9 +33,6 @@ def test_float_loop_direct(self): py.test.skip('fixme') - def test_large_switch_direct(self): - py.test.skip('no promotion/flexswitch for now please :-)') - def test_defaultonly_switch(self): py.test.skip('no promotion/flexswitch for now please :-)') @@ -125,9 +122,6 @@ def test_largedummy_compile(self): py.test.skip('it works only if we increase .maxstack') - def test_large_switch_compile(self): - py.test.skip('no promotion/flexswitch for now please :-)') - def test_read_frame_var_compile(self): py.test.skip('fixme: add support for frames') Modified: pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs (original) +++ pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Thu Jul 24 14:39:47 2008 @@ -116,8 +116,8 @@ { public int default_blockid = -1; public int numcases = 0; - public int[] values = new int[10]; // XXX maxlength? - public FlexSwitchCase[] cases = new FlexSwitchCase[10]; + public int[] values = new int[4]; + public FlexSwitchCase[] cases = new FlexSwitchCase[4]; public void set_default_blockid(int blockid) { @@ -126,10 +126,24 @@ public void add_case(int v, FlexSwitchCase c) { + if (numcases >= values.Length) + grow(); values[numcases] = v; cases[numcases] = c; numcases++; } + + private void grow() + { + int newsize = values.Length * 2; + int[] newvalues = new int[newsize]; + Array.Copy(values, newvalues, values.Length); + values = newvalues; + + FlexSwitchCase[] newcases = new FlexSwitchCase[newsize]; + Array.Copy(cases, newcases, cases.Length); + cases = newcases; + } public int execute(int v, InputArgs args) { From antocuni at codespeak.net Thu Jul 24 14:55:48 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 24 Jul 2008 14:55:48 +0200 (CEST) Subject: [pypy-svn] r56772 - pypy/branch/oo-jit/pypy/jit/codegen/cli/test Message-ID: <20080724125548.96B96169EF9@codespeak.net> Author: antocuni Date: Thu Jul 24 14:55:46 2008 New Revision: 56772 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Log: this test passes out of the box Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/test/test_rgenop.py Thu Jul 24 14:55:46 2008 @@ -33,9 +33,6 @@ def test_float_loop_direct(self): py.test.skip('fixme') - def test_defaultonly_switch(self): - py.test.skip('no promotion/flexswitch for now please :-)') - def test_read_frame_var_direct(self): py.test.skip('fixme: add support for frames') From antocuni at codespeak.net Thu Jul 24 17:29:51 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 24 Jul 2008 17:29:51 +0200 (CEST) Subject: [pypy-svn] r56775 - in pypy/branch/oo-jit/pypy: jit/codegen/cli translator/cli/src Message-ID: <20080724152951.E7F3D1684FF@codespeak.net> Author: antocuni Date: Thu Jul 24 17:29:51 2008 New Revision: 56775 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Log: pass object and do a cast inside cases instead of passing directly InputArgs; this will allow us to choose the exact class to use for inputargs depending on which field we need Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Thu Jul 24 17:29:51 2008 @@ -370,6 +370,7 @@ self.graphinfo = graphinfo self.gv_entrypoint = FunctionConst(delegatetype) + self.gv_inputargs = None self.genconsts = {} self.branches = [] self.newbranch() @@ -452,7 +453,6 @@ return self.graphinfo.has_flexswitches = True self.il_dispatch_jump_label = self.il.DefineLabel() - self.gv_inputargs = self.newlocalvar(class2type(cInputArgs)) self.jumpto_var = self.il.DeclareLocal(class2type(cInt32)) def get_op_Return(self, gv_returnvar): @@ -468,8 +468,10 @@ def emit_preamble(self): if not self.graphinfo.has_flexswitches: - return - # InputArgs inputargs_var = new InputArgs() + return + + # InputArgs inputargs = new InputArgs() + self.gv_inputargs = self.newlocalvar(class2type(cInputArgs)) clitype = self.gv_inputargs.getCliType() ctor = clitype.GetConstructor(new_array(System.Type, 0)) self.il.Emit(OpCodes.Newobj, ctor) @@ -522,8 +524,6 @@ gv_local = self.newlocalvar(gv_arg.getCliType()) self.linkargs_gv_map[gv_arg] = gv_local - self.gv_inputargs = self.inputargs_gv[1] # InputArgs is always the 2nd argument - def map_genvar(self, gv_var): return self.linkargs_gv_map.get(gv_var, gv_var) @@ -538,6 +538,14 @@ def emit_preamble(self): from pypy.jit.codegen.cli.operation import InputArgsManager + + # InputArgs inputargs = (InputArgs)obj // obj is the 2nd arg + clitype = class2type(cInputArgs) + self.gv_inputargs = self.newlocalvar(clitype) + self.inputargs_gv[1].load(self) + self.il.Emit(OpCodes.Castclass, clitype) + self.gv_inputargs.store(self) + linkargs_out_gv = [] for gv_linkarg in self.linkargs_gv: gv_var = self.linkargs_gv_map[gv_linkarg] @@ -677,7 +685,7 @@ graph = self.graph name = graph.name + '_case' restype = class2type(cInt32) - arglist = [class2type(cInt32), class2type(cInputArgs)] + arglist = [class2type(cInt32), class2type(cObject)] delegatetype = class2type(cFlexSwitchCase) graphinfo = graph.graphinfo meth = FlexSwitchCaseGenerator(graph.rgenop, name, restype, Modified: pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs ============================================================================== --- pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs (original) +++ pypy/branch/oo-jit/pypy/translator/cli/src/pypylib.cs Thu Jul 24 17:29:51 2008 @@ -106,7 +106,7 @@ { // XXX: these classes should be automatically generated by the JIT backend - public delegate int FlexSwitchCase(int block, InputArgs args); + public delegate int FlexSwitchCase(int block, object args); public class InputArgs { public int Int32_0; public int Int32_1; @@ -145,7 +145,7 @@ cases = newcases; } - public int execute(int v, InputArgs args) + public int execute(int v, object args) { for(int i=0; i Author: antocuni Date: Fri Jul 25 10:13:12 2008 New Revision: 56782 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Log: move InputArgsManager to rgenop.py, and rename copy_to_args and copy_from_args to avoid confusion Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Fri Jul 25 10:13:12 2008 @@ -142,9 +142,10 @@ return None def emit(self): + from pypy.jit.codegen.cli.rgenop import InputArgsManager il = self.meth.il manager = InputArgsManager(self.meth, [self.gv_x]) - manager.copy_from_args() + manager.copy_to_inputargs() blockid = self.meth.graphinfo.graph_retlabel.blockid il.Emit(OpCodes.Ldc_I4, blockid) il.Emit(OpCodes.Ret) @@ -224,6 +225,7 @@ return None def emit(self): + from pypy.jit.codegen.cli.rgenop import InputArgsManager graph = self.meth il = graph.il # get MethodInfo for LowLevelFlexSwitch.execute @@ -232,7 +234,7 @@ # setup the correct inputargs manager = InputArgsManager(graph, self.args_gv) - manager.copy_from_args() + manager.copy_to_inputargs() # jumpto = flexswitch.execute(exitswitch, inputargs); # goto dispatch_jump; @@ -244,54 +246,6 @@ il.Emit(OpCodes.Br, graph.il_dispatch_jump_label) -class InputArgsManager: - - def __init__(self, meth, args_gv): - self.meth = meth - self.args_gv = args_gv - - def basename_from_type(self, clitype): - return clitype.get_Name() - - def _get_fields(self): - fields = [] - gv_inputargs = self.meth.gv_inputargs - inputargs_clitype = gv_inputargs.getCliType() - counters = {} - for gv_arg in self.args_gv: - clitype = gv_arg.getCliType() - basename = self.basename_from_type(clitype) - count = counters.get(clitype, 0) - fieldname = '%s_%d' % (basename, count) - counters[clitype] = count+1 - field = inputargs_clitype.GetField(fieldname) - fields.append(field) - return fields - - def copy_from_args(self): - il = self.meth.il - gv_inputargs = self.meth.gv_inputargs - fields = self._get_fields() - assert len(self.args_gv) == len(fields) - for i in range(len(self.args_gv)): - gv_arg = self.args_gv[i] - field = fields[i] - gv_inputargs.load(self.meth) - gv_arg.load(self.meth) - il.Emit(OpCodes.Stfld, field) - - def copy_to_args(self): - il = self.meth.il - gv_inputargs = self.meth.gv_inputargs - fields = self._get_fields() - assert len(self.args_gv) == len(fields) - for i in range(len(self.args_gv)): - gv_arg = self.args_gv[i] - field = fields[i] - gv_inputargs.load(self.meth) - il.Emit(OpCodes.Ldfld, field) - gv_arg.store(self.meth) - class WriteLine(Operation): def __init__(self, meth, message): Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Fri Jul 25 10:13:12 2008 @@ -240,12 +240,11 @@ self.inputargs_gv = inputargs_gv def emit_trampoline(self, meth): - from pypy.jit.codegen.cli.operation import InputArgsManager from pypy.jit.codegen.cli.operation import mark il = meth.il manager = InputArgsManager(meth, self.inputargs_gv) il.MarkLabel(self.il_trampoline_label) - manager.copy_to_args() + manager.copy_from_inputargs() il.Emit(OpCodes.Br, self.il_label) class RCliGenOp(AbstractRGenOp): @@ -537,8 +536,6 @@ self.parent_flexswitch.llflexswitch.add_case(self.value, func2) def emit_preamble(self): - from pypy.jit.codegen.cli.operation import InputArgsManager - # InputArgs inputargs = (InputArgs)obj // obj is the 2nd arg clitype = class2type(cInputArgs) self.gv_inputargs = self.newlocalvar(clitype) @@ -551,7 +548,7 @@ gv_var = self.linkargs_gv_map[gv_linkarg] linkargs_out_gv.append(gv_var) manager = InputArgsManager(self, linkargs_out_gv) - manager.copy_to_args() + manager.copy_from_inputargs() class BranchBuilder(GenBuilder): @@ -697,6 +694,54 @@ return meth.branches[0] +class InputArgsManager: + + def __init__(self, meth, args_gv): + self.meth = meth + self.args_gv = args_gv + + def basename_from_type(self, clitype): + return clitype.get_Name() + + def _get_fields(self): + fields = [] + gv_inputargs = self.meth.gv_inputargs + inputargs_clitype = gv_inputargs.getCliType() + counters = {} + for gv_arg in self.args_gv: + clitype = gv_arg.getCliType() + basename = self.basename_from_type(clitype) + count = counters.get(clitype, 0) + fieldname = '%s_%d' % (basename, count) + counters[clitype] = count+1 + field = inputargs_clitype.GetField(fieldname) + fields.append(field) + return fields + + def copy_to_inputargs(self): + il = self.meth.il + gv_inputargs = self.meth.gv_inputargs + fields = self._get_fields() + assert len(self.args_gv) == len(fields) + for i in range(len(self.args_gv)): + gv_arg = self.args_gv[i] + field = fields[i] + gv_inputargs.load(self.meth) + gv_arg.load(self.meth) + il.Emit(OpCodes.Stfld, field) + + def copy_from_inputargs(self): + il = self.meth.il + gv_inputargs = self.meth.gv_inputargs + fields = self._get_fields() + assert len(self.args_gv) == len(fields) + for i in range(len(self.args_gv)): + gv_arg = self.args_gv[i] + field = fields[i] + gv_inputargs.load(self.meth) + il.Emit(OpCodes.Ldfld, field) + gv_arg.store(self.meth) + global_rgenop = RCliGenOp() RCliGenOp.constPrebuiltGlobal = global_rgenop.genconst zero_const = ObjectConst(ootype.NULL) From antocuni at codespeak.net Fri Jul 25 14:34:32 2008 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 25 Jul 2008 14:34:32 +0200 (CEST) Subject: [pypy-svn] r56783 - pypy/branch/oo-jit/pypy/jit/codegen/cli Message-ID: <20080725123432.1E8662A8084@codespeak.net> Author: antocuni Date: Fri Jul 25 14:34:31 2008 New Revision: 56783 Added: pypy/branch/oo-jit/pypy/jit/codegen/cli/args_manager.py (contents, props changed) Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Log: dynamically compute the shape of InputArgs depending of which fields it actually needs; the final CLI type is obtained by instantiating one of the generic classes in pypylib. It still fails if you try to create InputArgs with too many arguments Added: pypy/branch/oo-jit/pypy/jit/codegen/cli/args_manager.py ============================================================================== --- (empty file) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/args_manager.py Fri Jul 25 14:34:31 2008 @@ -0,0 +1,130 @@ +from pypy.translator.cli.dotnet import CLR +from pypy.translator.cli import dotnet +System = CLR.System +Assembly = System.Reflection.Assembly +OpCodes = System.Reflection.Emit.OpCodes + +class ArgsManager: + + def __init__(self): + self.type_counter = {} + self.clitype = None + self.fieldtypes = None + self.slots = None + + def is_open(self): + return self.clitype is None + + def getCliType(self): + assert not self.is_open() + return self.clitype + + def register(self, args_gv): + assert self.is_open() + newcounter = {} + for gv_arg in args_gv: + clitype = gv_arg.getCliType() + newcount = newcounter.get(clitype, 0) + newcounter[clitype] = newcount+1 + + for clitype, newcount in newcounter.iteritems(): + oldcount = self.type_counter.get(clitype, 0) + maxcount = max(oldcount, newcount) + self.type_counter[clitype] = maxcount + + def close(self): + assert self.is_open() + templates = self._get_templates() + + self.fieldtypes = fieldtypes = [] + self.slots = slots = {} + for clitype, count in self.type_counter.iteritems(): + start = len(fieldtypes) + end = start+count + fieldtypes += [clitype]*count + slots[clitype] = self._myrange(start, end) + numfields = len(fieldtypes) + + if numfields <= len(templates): + template = templates[numfields-1] + array = dotnet.new_array(System.Type, numfields) + for i in range(numfields): + array[i] = fieldtypes[i] + self.clitype = template.MakeGenericType(array) + else: + assert False, 'TODO' + + def copy_to_inputargs(self, meth, args_gv): + "copy args_gv into the appropriate fields of inputargs" + assert not self.is_open() + il = meth.il + gv_inputargs = meth.gv_inputargs + fields = self._get_fields(args_gv) + assert len(args_gv) == len(fields) + for i in range(len(args_gv)): + gv_arg = args_gv[i] + fieldinfo = fields[i] + gv_inputargs.load(meth) + gv_arg.load(meth) + il.Emit(OpCodes.Stfld, fieldinfo) + + def copy_from_inputargs(self, meth, args_gv): + "copy the appropriate fields of inputargs into args_gv" + assert not self.is_open() + il = meth.il + gv_inputargs = meth.gv_inputargs + fields = self._get_fields(args_gv) + assert len(args_gv) == len(fields) + for i in range(len(args_gv)): + gv_arg = args_gv[i] + fieldinfo = fields[i] + gv_inputargs.load(meth) + il.Emit(OpCodes.Ldfld, fieldinfo) + gv_arg.store(meth) + + def _myrange(self, start, end): + length = (end - start) + res = [0] * length + for i in range(start, end): + res[i] = i + return res + + def _load_pypylib(self): + from pypy.translator.cli.query import pypylib, pypylib2 + assembly = None + for name in [pypylib, pypylib2]: + assembly = Assembly.LoadWithPartialName(name) + if assembly: + break + assert assembly is not None + return assembly + + def _get_templates(self): + pypylib = self._load_pypylib() + templates = [] + i = 1 + while True: + typename = 'pypy.runtime.InputArgs`%d' % i + clitype = pypylib.GetType(typename) + if not clitype: + break + templates.append(clitype) + i += 1 + return templates + + def _copy_slots(self): + 'Deepcopy self.slots' + slots = {} + for key, value in self.slots.iteritems(): + slots[key] = value[:] + return slots + + def _get_fields(self, args_gv): + slots = self._copy_slots() + types = [gv_arg.getCliType() for gv_arg in args_gv] + fields = [] + for clitype in types: + slot = slots[clitype].pop() + fieldinfo = self.clitype.GetField('field%d' % slot) + fields.append(fieldinfo) + return fields Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/operation.py Fri Jul 25 14:34:31 2008 @@ -142,11 +142,10 @@ return None def emit(self): - from pypy.jit.codegen.cli.rgenop import InputArgsManager il = self.meth.il - manager = InputArgsManager(self.meth, [self.gv_x]) - manager.copy_to_inputargs() - blockid = self.meth.graphinfo.graph_retlabel.blockid + graphinfo = self.meth.graphinfo + graphinfo.args_manager.copy_to_inputargs(self.meth, [self.gv_x]) + blockid = graphinfo.graph_retlabel.blockid il.Emit(OpCodes.Ldc_I4, blockid) il.Emit(OpCodes.Ret) @@ -225,7 +224,6 @@ return None def emit(self): - from pypy.jit.codegen.cli.rgenop import InputArgsManager graph = self.meth il = graph.il # get MethodInfo for LowLevelFlexSwitch.execute @@ -233,8 +231,8 @@ meth_execute = clitype.GetMethod('execute') # setup the correct inputargs - manager = InputArgsManager(graph, self.args_gv) - manager.copy_to_inputargs() + args_manager = graph.graphinfo.args_manager + args_manager.copy_to_inputargs(graph, self.args_gv) # jumpto = flexswitch.execute(exitswitch, inputargs); # goto dispatch_jump; Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/cli/rgenop.py Fri Jul 25 14:34:31 2008 @@ -8,6 +8,7 @@ from pypy.jit.codegen.model import CodeGenSwitch from pypy.jit.codegen.cli import operation as ops from pypy.jit.codegen.cli.methodfactory import get_method_wrapper +from pypy.jit.codegen.cli.args_manager import ArgsManager from pypy.translator.cli.dotnet import CLR, typeof, new_array, init_array from pypy.translator.cli.dotnet import box, unbox, clidowncast, classof from pypy.translator.cli import dotnet @@ -15,7 +16,6 @@ DelegateHolder = CLR.pypy.runtime.DelegateHolder LowLevelFlexSwitch = CLR.pypy.runtime.LowLevelFlexSwitch FlexSwitchCase = CLR.pypy.runtime.FlexSwitchCase -InputArgs = CLR.pypy.runtime.InputArgs OpCodes = System.Reflection.Emit.OpCodes cVoid = ootype.nullruntimeclass @@ -25,7 +25,6 @@ cObject = classof(System.Object) cString = classof(System.String) cChar = classof(System.Char) -cInputArgs = classof(InputArgs) cUtils = classof(CLR.pypy.runtime.Utils) cFlexSwitchCase = classof(FlexSwitchCase) @@ -242,9 +241,9 @@ def emit_trampoline(self, meth): from pypy.jit.codegen.cli.operation import mark il = meth.il - manager = InputArgsManager(meth, self.inputargs_gv) + args_manager = meth.graphinfo.args_manager il.MarkLabel(self.il_trampoline_label) - manager.copy_from_inputargs() + args_manager.copy_from_inputargs(meth, self.inputargs_gv) il.Emit(OpCodes.Br, self.il_label) class RCliGenOp(AbstractRGenOp): @@ -351,6 +350,7 @@ self.blocks = [] # blockid -> (meth, label) self.flexswitch_meths = [] self.main_retlabel = None + self.args_manager = ArgsManager() class MethodGenerator: @@ -397,6 +397,7 @@ blockid = len(blocks) result = Label(blockid, self.il, args_gv) blocks.append((self, result)) + self.graphinfo.args_manager.register(args_gv) return result def newlocalvar(self, clitype): @@ -458,8 +459,8 @@ return ops.Return(self, gv_returnvar) def emit_code(self): - self.emit_flexswitches() MethodGenerator.emit_code(self) + self.emit_flexswitches() def emit_flexswitches(self): for meth in self.graphinfo.flexswitch_meths: @@ -468,9 +469,13 @@ def emit_preamble(self): if not self.graphinfo.has_flexswitches: return - + + # compute the shape of InputArgs + args_manager = self.graphinfo.args_manager + args_manager.close() + # InputArgs inputargs = new InputArgs() - self.gv_inputargs = self.newlocalvar(class2type(cInputArgs)) + self.gv_inputargs = self.newlocalvar(args_manager.getCliType()) clitype = self.gv_inputargs.getCliType() ctor = clitype.GetConstructor(new_array(System.Type, 0)) self.il.Emit(OpCodes.Newobj, ctor) @@ -522,6 +527,7 @@ for gv_arg in linkargs_gv: gv_local = self.newlocalvar(gv_arg.getCliType()) self.linkargs_gv_map[gv_arg] = gv_local + self.graphinfo.args_manager.register(linkargs_gv) def map_genvar(self, gv_var): return self.linkargs_gv_map.get(gv_var, gv_var) @@ -537,7 +543,8 @@ def emit_preamble(self): # InputArgs inputargs = (InputArgs)obj // obj is the 2nd arg - clitype = class2type(cInputArgs) + args_manager = self.graphinfo.args_manager + clitype = args_manager.getCliType() self.gv_inputargs = self.newlocalvar(clitype) self.inputargs_gv[1].load(self) self.il.Emit(OpCodes.Castclass, clitype) @@ -547,9 +554,7 @@ for gv_linkarg in self.linkargs_gv: gv_var = self.linkargs_gv_map[gv_linkarg] linkargs_out_gv.append(gv_var) - manager = InputArgsManager(self, linkargs_out_gv) - manager.copy_from_inputargs() - + args_manager.copy_from_inputargs(self, linkargs_out_gv) class BranchBuilder(GenBuilder): @@ -693,55 +698,6 @@ meth.set_linkargs_gv(self.linkargs_gv) return meth.branches[0] - -class InputArgsManager: - - def __init__(self, meth, args_gv): - self.meth = meth - self.args_gv = args_gv - - def basename_from_type(self, clitype): - return clitype.get_Name() - - def _get_fields(self): - fields = [] - gv_inputargs = self.meth.gv_inputargs - inputargs_clitype = gv_inputargs.getCliType() - counters = {} - for gv_arg in self.args_gv: - clitype = gv_arg.getCliType() - basename = self.basename_from_type(clitype) - count = counters.get(clitype, 0) - fieldname = '%s_%d' % (basename, count) - counters[clitype] = count+1 - field = inputargs_clitype.GetField(fieldname) - fields.append(field) - return fields - - def copy_to_inputargs(self): - il = self.meth.il - gv_inputargs = self.meth.gv_inputargs - fields = self._get_fields() - assert len(self.args_gv) == len(fields) - for i in range(len(self.args_gv)): - gv_arg = self.args_gv[i] - field = fields[i] - gv_inputargs.load(self.meth) - gv_arg.load(self.meth) - il.Emit(OpCodes.Stfld, field) - - def copy_from_inputargs(self): - il = self.meth.il - gv_inputargs = self.meth.gv_inputargs - fields = self._get_fields() - assert len(self.args_gv) == len(fields) - for i in range(len(self.args_gv)): - gv_arg = self.args_gv[i] - field = fields[i] - gv_inputargs.load(self.meth) - il.Emit(OpCodes.Ldfld, field) - gv_arg.store(self.meth) - global_rgenop = RCliGenOp() RCliGenOp.constPrebuiltGlobal = global_rgenop.genconst zero_const = ObjectConst(ootype.NULL) From witulski at codespeak.net Wed Jul 30 14:17:55 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Wed, 30 Jul 2008 14:17:55 +0200 (CEST) Subject: [pypy-svn] r56858 - in pypy/branch/oo-jit/pypy/jit/codegen: . x86_64 x86_64/test Message-ID: <20080730121755.63EBB4981EE@codespeak.net> Author: witulski Date: Wed Jul 30 14:17:54 2008 New Revision: 56858 Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/__init__.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/__init__.py Modified: pypy/branch/oo-jit/pypy/jit/codegen/detect_cpu.py Log: (cfbolz,witulski) added new empty x86_64 folder. fixed autodetect Modified: pypy/branch/oo-jit/pypy/jit/codegen/detect_cpu.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/detect_cpu.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/detect_cpu.py Wed Jul 30 14:17:54 2008 @@ -22,8 +22,11 @@ mach = os.popen('uname -m', 'r').read().strip() if not mach: raise ProcessorAutodetectError, "cannot run 'uname -m'" - if mach == 'x86_64' and sys.maxint == 2147483647: - mach = 'x86' # it's a 64-bit processor but in 32-bits mode, maybe + if mach == 'x86_64': + if sys.maxint == 2147483647: + mach = 'x86' # it's a 64-bit processor but in 32-bits mode, maybe + else: + assert sys.maxint == 2 ** 63 - 1 try: return {'i386': 'i386', 'i486': 'i386', @@ -31,7 +34,8 @@ 'i686': 'i386', 'i86pc': 'i386', # Solaris/Intel 'x86': 'i386', # Apple - 'Power Macintosh': 'ppc', + 'Power Macintosh': 'ppc', + 'x86_64': 'x86_64', }[mach] except KeyError: raise ProcessorAutodetectError, "unsupported processor '%s'" % mach Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/__init__.py ============================================================================== Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/__init__.py ============================================================================== From arigo at codespeak.net Wed Jul 30 19:29:37 2008 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 30 Jul 2008 19:29:37 +0200 (CEST) Subject: [pypy-svn] r56867 - pypy/dist/pypy/doc/jit Message-ID: <20080730172937.BE228168452@codespeak.net> Author: arigo Date: Wed Jul 30 19:29:36 2008 New Revision: 56867 Modified: pypy/dist/pypy/doc/jit/index.txt Log: Link to the blog post for now. Modified: pypy/dist/pypy/doc/jit/index.txt ============================================================================== --- pypy/dist/pypy/doc/jit/index.txt (original) +++ pypy/dist/pypy/doc/jit/index.txt Wed Jul 30 19:29:36 2008 @@ -29,8 +29,12 @@ - Machine code Backends_ +- Current work and prototype in Prolog (!): see `this blog post`__. + *(work and documentation in progress!)* +.. __: http://morepypy.blogspot.com/2008/06/hi-all-some-news-from-jit-front.html + ------------------------------------------------------------ .. _Overview: overview.html From cami at codespeak.net Wed Jul 30 21:21:11 2008 From: cami at codespeak.net (cami at codespeak.net) Date: Wed, 30 Jul 2008 21:21:11 +0200 (CEST) Subject: [pypy-svn] r56868 - in pypy/dist/pypy/lang/gameboy: . debug test Message-ID: <20080730192111.9E536168532@codespeak.net> Author: cami Date: Wed Jul 30 21:21:10 2008 New Revision: 56868 Added: pypy/dist/pypy/lang/gameboy/debug/ pypy/dist/pypy/lang/gameboy/debug/gameboy_debug_entry_point.py (contents, props changed) Removed: pypy/dist/pypy/lang/gameboy/gameboyTest.py Modified: pypy/dist/pypy/lang/gameboy/cpu.py pypy/dist/pypy/lang/gameboy/gameboy_implementation.py pypy/dist/pypy/lang/gameboy/test/test_cpu.py Log: created debug implementation to compare ram and register with the original java version remove debug statements from the other version (cpu, gameboy_implementation) Modified: pypy/dist/pypy/lang/gameboy/cpu.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/cpu.py (original) +++ pypy/dist/pypy/lang/gameboy/cpu.py Wed Jul 30 21:21:10 2008 @@ -2,7 +2,10 @@ from pypy.lang.gameboy import constants from pypy.lang.gameboy.ram import * from pypy.lang.gameboy.interrupt import * -#from pypy.lang.gameboy.debug import * + +from pypy.rlib.objectmodel import we_are_translated +if not we_are_translated(): + from pypy.lang.gameboy.debug import * # --------------------------------------------------------------------------- @@ -365,7 +368,7 @@ if self.interrupt.is_pending(): self.halted = False self.cycles -= 4 - elif (self.cycles > 0): + elif self.cycles > 0: self.cycles = 0 def lower_pending_interrupt(self): @@ -377,24 +380,15 @@ return def fetch_execute(self): - opCode = self.fetch() - #print " fetch exe:", hex(opCode), " " - #, FETCH_EXECUTE_OP_CODES[opCode].__name__ - self.last_fetch_execute_op_code = opCode - #if DEBUG: log(opCode, is_fetch_execute=True) - FETCH_EXECUTE_OP_CODES[opCode](self) + op_code = self.fetch() + self.last_fetch_execute_op_code = op_code + FETCH_EXECUTE_OP_CODES[op_code](self) - def execute(self, opCode): + def execute(self, op_code): self.instruction_counter += 1 - #if DEBUG: log(opCode) - #print self.instruction_counter, "-"*60 - #print "exe: ", hex(opCode), " " - #, OP_CODES[opCode].__name__ - #print " pc:", hex(self.pc.get()), "sp:", hex(self.sp.get()) - #self.print_registers() - self.last_op_code = opCode - OP_CODES[opCode](self) + self.last_op_code = op_code + OP_CODES[op_code](self) def print_registers(self): print " a: "+hex(self.a.get()) @@ -892,7 +886,7 @@ # RETI 4 cycles self.ret() # 4 cycles self.enable_interrupts() # 1 cycle + others - self.cycles += 1 + #self.cycles += 1 def restart(self, nn): # RST nn 4 cycles @@ -983,35 +977,35 @@ def get(self, use_cycles=True): return self.cpu.fetch(use_cycles) -# OPCODE LOOKUP TABLE GENERATION ----------------------------------------------- +# op_code LOOKUP TABLE GENERATION ----------------------------------------------- GROUPED_REGISTERS = [CPU.get_b, CPU.get_c, CPU.get_d, CPU.get_e, CPU.get_h, CPU.get_l, CPU.get_hli, CPU.get_a] def create_group_op_codes(table): - opCodes =[] + op_codes =[] for entry in table: - opCode = entry[0] + op_code = entry[0] step = entry[1] function = entry[2] if len(entry) == 4: for registerGetter in GROUPED_REGISTERS: for n in entry[3]: - opCodes.append((opCode, group_lambda(function, registerGetter, n))) - opCode += step + op_codes.append((op_code, group_lambda(function, registerGetter, n))) + op_code += step if len(entry) == 5: entryStep = entry[4] for registerGetter in GROUPED_REGISTERS: - stepOpCode = opCode + stepop_code = op_code for n in entry[3]: - opCodes.append((stepOpCode, group_lambda(function, registerGetter, n))) - stepOpCode += entryStep - opCode+=step + op_codes.append((stepop_code, group_lambda(function, registerGetter, n))) + stepop_code += entryStep + op_code+=step else: for registerGetter in GROUPED_REGISTERS: - opCodes.append((opCode,group_lambda(function, registerGetter))) - opCode += step - return opCodes + op_codes.append((op_code,group_lambda(function, registerGetter))) + op_code += step + return op_codes def group_lambda(function, register_getter, value=None): if value is None: @@ -1022,29 +1016,29 @@ RegisterCallWrapper(register_getter(s)), value) def create_load_group_op_codes(): - opCodes = [] - opCode = 0x40 + op_codes = [] + op_code = 0x40 for storeRegister in GROUPED_REGISTERS: for loadRegister in GROUPED_REGISTERS: if loadRegister != CPU.get_hli or storeRegister != CPU.get_hli: - opCodes.append((opCode, load_group_lambda(storeRegister, loadRegister))) - opCode += 1 - return opCodes + op_codes.append((op_code, load_group_lambda(storeRegister, loadRegister))) + op_code += 1 + return op_codes def load_group_lambda(store_register, load_register): return lambda s: CPU.ld(s, RegisterCallWrapper(load_register(s)), RegisterCallWrapper(store_register(s))) def create_register_op_codes(table): - opCodes = [] + op_codes = [] for entry in table: - opCode = entry[0] + op_code = entry[0] step = entry[1] function = entry[2] for registerOrGetter in entry[3]: - opCodes.append((opCode, register_lambda(function, registerOrGetter))) - opCode += step - return opCodes + op_codes.append((op_code, register_lambda(function, registerOrGetter))) + op_code += step + return op_codes def register_lambda(function, registerOrGetter): if callable(registerOrGetter): @@ -1066,7 +1060,7 @@ result[pos] = entry[-1] return result -# OPCODE TABLES --------------------------------------------------------------- +# op_code TABLES --------------------------------------------------------------- # Table with one to one mapping of simple OP Codes FIRST_ORDER_OP_CODES = [ (0x00, CPU.nop), @@ -1147,7 +1141,7 @@ REGISTER_SET_B = [CPU.get_bc, CPU.get_de, CPU.get_hl, CPU.get_af] FLAG_REGISTER_SET = [CPU.is_not_z, CPU.is_z, CPU.is_not_c, CPU.is_c] -# Table for Register OP Codes: (startAddress, delta, method, regsiters) +# Table for Register OP Codes: (startAddress, delta, method, registers) REGISTER_OP_CODES = [ (0x01, 0x10, CPU.fetch_double_register, REGISTER_SET_A), (0x09, 0x10, CPU.add_hl, REGISTER_SET_A), @@ -1160,7 +1154,7 @@ (0xC1, 0x10, CPU.pop_double_register, REGISTER_SET_B), (0xC5, 0x10, CPU.push_double_register, REGISTER_SET_B) ] -# Table for Second Order OPCodes: (startAddress, delta, method, [args]) +# Table for Second Order op_codes: (startAddress, delta, method, [args]) SECOND_ORDER_REGISTER_GROUP_OP_CODES = [ (0x00, 0x01, CPU.rotate_left_circular), (0x08, 0x01, CPU.rotate_right_circular), @@ -1175,7 +1169,7 @@ (0x80, 0x01, CPU.reset_bit, range(0, 8), 0x08) ] -# RAW OPCODE TABLE INITIALIZATION ---------------------------------------------- +# RAW op_code TABLE INITIALIZATION ---------------------------------------------- FIRST_ORDER_OP_CODES += create_register_op_codes(REGISTER_OP_CODES) FIRST_ORDER_OP_CODES += create_group_op_codes(REGISTER_GROUP_OP_CODES) Added: pypy/dist/pypy/lang/gameboy/debug/gameboy_debug_entry_point.py ============================================================================== --- (empty file) +++ pypy/dist/pypy/lang/gameboy/debug/gameboy_debug_entry_point.py Wed Jul 30 21:21:10 2008 @@ -0,0 +1,28 @@ +from pypy.lang.gameboy.debug.gameboy_debug_implementation import * +import py +import sys + +from AppKit import NSApplication +NSApplication.sharedApplication() + +ROM_PATH = str(py.magic.autopath().dirpath().dirpath())+"/rom" + +filename = "" +if len(sys.argv) > 1: + print sys.argv + filename = sys.argv[1] +else: + pos = str(9) + filename = ROM_PATH+"/rom"+pos+"/rom"+pos+".gb" +print "loading rom: ", str(filename) + +gameBoy = GameBoyDebugImplementation() + +try: + gameBoy.load_cartridge_file(str(filename)) +except: + gameBoy.load_cartridge_file(str(filename), verify=False) + print "Cartridge is Corrupted!" + +gameBoy.mainLoop() + Modified: pypy/dist/pypy/lang/gameboy/gameboy_implementation.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/gameboy_implementation.py (original) +++ pypy/dist/pypy/lang/gameboy/gameboy_implementation.py Wed Jul 30 21:21:10 2008 @@ -6,12 +6,10 @@ from pypy.lang.gameboy.sound import SoundDriver from pypy.lang.gameboy.timer import Clock from pypy.lang.gameboy import constants -#from pypy.lang.gameboy import debug from pypy.rlib.rsdl import RSDL, RSDL_helper from pypy.rpython.lltypesystem import lltype, rffi from pypy.rlib.objectmodel import specialize -import py # GAMEBOY ---------------------------------------------------------------------- @@ -41,9 +39,12 @@ finally: lltype.free(self.event, flavor='raw') RSDL.Quit() - #debug.print_results() + self.handle_execution_error() return 0 + def handle_execution_error(self): + pass + def handle_events(self): isRunning = True while self.poll_event(): @@ -93,7 +94,6 @@ #if y%2 == 0 or True: # px = self.get_pixel_color(x, y) # str += ["#", "%", "+", " ", " "][px] - RSDL_helper.set_pixel(self.screen, x, y, self.pixel_map(x, y)) pass #print str; Modified: pypy/dist/pypy/lang/gameboy/test/test_cpu.py ============================================================================== --- pypy/dist/pypy/lang/gameboy/test/test_cpu.py (original) +++ pypy/dist/pypy/lang/gameboy/test/test_cpu.py Wed Jul 30 21:21:10 2008 @@ -356,32 +356,37 @@ assert_default_registers(cpu, pc=pc+1) # jr_nn -def test_0x18(): +def test_0x18_relative_jump(): cpu = get_cpu(); - pc = cpu.pc.get() - value = 0x12 - cpu.rom[constants.RESET_PC] = value - assert_default_registers(cpu) - cycle_test(cpu, 0x18, 3) - assert_default_registers(cpu, pc=pc+value+1) + for jump in range(-128,128): + cpu.pc.set(0x1234) + prepare_for_fetch(cpu, jump & 0xFF) + cycle_test(cpu, 0x18, 3) + # relative offset + one fetch + assert cpu.pc.get() == 0x1234 + jump + 1 + assert_default_registers(cpu, f=cpu.f.get(), pc=0x1234+jump+1) # jr_NZ_nn see test_jr_cc_nn -def test_0x20_0x28_0x30(): +def test_0x20_to_0x38_relative_conditional_jump(): cpu = get_cpu() flags = [~constants.Z_FLAG, constants.Z_FLAG, ~constants.C_FLAG, constants.C_FLAG] opCode = 0x20 value = 0x12 for i in range(0, 4): - prepare_for_fetch(cpu, value) - pc = cpu.pc.get() - cpu.f.set(flags[i]) - cycle_test(cpu, opCode, 3) - assert cpu.pc.get() == pc+value+1 + for jump in range(-128,128): + cpu.pc.set(0x1234) + prepare_for_fetch(cpu, jump & 0xFF) + cpu.f.set(flags[i]) + cycle_test(cpu, opCode, 3) + # relative offset + one fetch + assert cpu.pc.get() == 0x1234 + jump + 1 + assert_default_registers(cpu, f=cpu.f.get(), pc=0x1234+jump+1) pc = cpu.pc.get() cpu.f.set(~flags[i]) cycle_test(cpu, opCode, 2) assert cpu.pc.get() == pc+1 + assert_default_registers(cpu, f=cpu.f.get(), pc=pc+1) value += 3 opCode += 0x08 @@ -443,7 +448,7 @@ assert cpu.read(cpu.de.get()) == cpu.a.get() # load_a_DEi -def test_0x1A(): +def test_0x1A_store_memory_at_de_in_a(): cpu = get_cpu() value = 0x12 address = 0xC020 @@ -1056,7 +1061,7 @@ assert_default_registers(cpu, a=value, pc=pc+2) # ld_mem_A -def test_0xEA(): +def test_0xEA_store_a_at_fetched_address(): cpu = get_cpu() valueA = 0x56 prepare_for_fetch(cpu, 0x12, 0x34) @@ -1110,7 +1115,8 @@ prepare_for_pop(cpu, value >> 8, value & 0xFF) prepare_for_fetch(cpu, 0x00) pc = cpu.pc.get() - cycle_test(cpu, 0xD9, 4+2) + cycle_test(cpu, 0xD9, 4+1+1) + # pc: popped value + assert_default_registers(cpu, pc=value+1, sp=2) def test_handle_interrupt(): @@ -1194,7 +1200,7 @@ opCode += 0x08 # ldh_Ci_A -def test_0xE2_write_a_at_expaded_c_address(): +def test_0xE2_write_a_at_expanded_c_address(): cpu = get_cpu() value = 0x12 valueA = value+1 @@ -1214,14 +1220,6 @@ cycle_test(cpu, 0xF2, 2) assert_default_registers(cpu, a=valueA, bc=valueC) -# jp_nnnn -def test_0xC3(): - cpu = get_cpu() - prepare_for_fetch(cpu, 0x12, 0x34) - cycle_test(cpu, 0xC3, 4) - assert_default_registers(cpu, pc=0x1234) - - # di def test_0xF3_disable_interrupt(): cpu = get_cpu() @@ -1269,6 +1267,15 @@ assert cpu.interrupt.serial.is_pending() == False assert cpu.interrupt.is_pending() == False +# jp_nnnn + +# JUMP AND CALL TESTING ======================================================== +def test_0xC3_jump(): + cpu = get_cpu() + prepare_for_fetch(cpu, 0x12, 0x34) + cycle_test(cpu, 0xC3, 4) + assert_default_registers(cpu, pc=0x1234) + def conditional_call_test(cpu, opCode, flagSetter): # set the condition to false and dont call flagSetter(cpu, False) @@ -1323,30 +1330,35 @@ def set_flag_0xDC(cpu, value): cpu.f.c_flag = value +# call_nnnn +def test_0xCD_call(): + cpu = get_cpu() + fetchValue = 0x1234 + cpu.sp.set(fetchValue) + prepare_for_fetch(cpu, fetchValue) + cycle_test(cpu, 0xCD, 6) + assert_default_registers(cpu, pc=fetchValue, sp=fetchValue-2) + # push_BC to push_AF -def test_0xC5_to_0xF5_push(): +def test_0xC5_to_0xF5_push_double_register(): cpu = get_cpu() registers = [cpu.bc, cpu.de, cpu.hl, cpu.af] opCode = 0xC5 value = 0x1234 for register in registers: + sp = cpu.sp.get() register.set(value) cycle_test(cpu, opCode, 4) assert cpu.memory.read(cpu.sp.get()+1) == value >> 8 - assert cpu.memory.read(cpu.sp.get()) == value & 0xFF + assert cpu.memory.read(cpu.sp.get()) == value & 0xFF + assert cpu.sp.get() == sp - 2 + assert cpu.pop() == value & 0xFF + assert cpu.pop() == value >> 8 + assert cpu.sp.get() == sp opCode += 0x10 value += 0x0101 - - -# call_nnnn -def test_0xCD_call(): - cpu = get_cpu() - fetchValue = 0x1234 - cpu.sp.set(fetchValue) - prepare_for_fetch(cpu, fetchValue) - cycle_test(cpu, 0xCD, 6) - assert_default_registers(cpu, pc=fetchValue, sp=fetchValue-2) - + + def a_nn_test(opCode, cycles, opCaller): # flags tested already cpu = get_cpu() From cami at codespeak.net Wed Jul 30 23:34:15 2008 From: cami at codespeak.net (cami at codespeak.net) Date: Wed, 30 Jul 2008 23:34:15 +0200 (CEST) Subject: [pypy-svn] r56869 - in pypy/dist/pypy/lang/gameboy: . debug Message-ID: <20080730213415.18F351684D3@codespeak.net> Author: cami Date: Wed Jul 30 23:34:14 2008 New Revision: 56869 Removed: pypy/dist/pypy/lang/gameboy/debug/ pypy/dist/pypy/lang/gameboy/debug.py Log: remove debug folder and debug.py since they belong to the java counterpart From witulski at codespeak.net Thu Jul 31 14:18:12 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Thu, 31 Jul 2008 14:18:12 +0200 (CEST) Subject: [pypy-svn] r56872 - in pypy/branch/oo-jit/pypy/jit/codegen/x86_64: . test Message-ID: <20080731121812.DC8F3168504@codespeak.net> Author: witulski Date: Thu Jul 31 14:18:11 2008 New Revision: 56872 Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Log: (cfBolz, witulski) Implemented encoding of ADD instruction for 64 bit registers. Needs to be refactored later. Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py ============================================================================== --- (empty file) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py Thu Jul 31 14:18:11 2008 @@ -0,0 +1,50 @@ +#Mapping from register to coding (Rex.W or Rex.B , ModRM) +REGISTER_MAP = { + "rax": (0, 0), + "rcx": (0, 1), + "rdx": (0, 2), + "rbx": (0, 3), + "rsp": (0, 4), + "rbp": (0, 5), + "rsi": (0, 6), + "rdi": (0, 7), + "r8": (1, 0), + "r9": (1, 1), + "r10": (1, 2), + "r11": (1, 3), + "r12": (1, 4), + "r13": (1, 5), + "r14": (1, 6), + "r15": (1, 7), + } + +class X86_64CodeBuilder(object): + """ creats x86_64 opcodes""" + def write(self, data): + """ writes data into memory""" + raise NotImplementedError + + def tell(self): + """ tells the current position in memory""" + raise NotImplementedError + + # XXX for now, arg1 and arg2 are registers + def ADD(self, arg1, arg2): + rexR, modrm1 = self.get_register_bits(arg1) + rexB, modrm2 = self.get_register_bits(arg2) + #rexW(1) = 64bitMode rexX(0) = doesn't matter + self.write_rex_byte(1, rexR, 0, rexB) + self.write('\x00') + self.write_modRM_byte(3, modrm1, modrm2) + + def get_register_bits(self, register): + return REGISTER_MAP[register] + + # Rex-Prefix 4WRXB see AMD vol3 page 45 + def write_rex_byte(self, rexW, rexR, rexX, rexB): + byte = (4 << 4) | (rexW << 3) | (rexR << 2) | (rexX << 1) | rexB + self.write(chr(byte)) + + def write_modRM_byte(self, mod, reg, rm): + byte = mod << 6 | (reg << 3) | rm + self.write(chr(byte)) \ No newline at end of file Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py ============================================================================== --- (empty file) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Thu Jul 31 14:18:11 2008 @@ -0,0 +1,16 @@ +from pypy.jit.codegen.x86_64 import assembler + +class AsmTest(assembler.X86_64CodeBuilder): + def __init__(self): + self.data = [] + + def get_as_string(self): + return "".join(self.data) + + def write(self,char): + self.data.append(char) + +def test_add(): + mc = AsmTest() + mc.ADD("rax", "r11") + assert mc.get_as_string() == "\x49\x00\xC3" \ No newline at end of file From witulski at codespeak.net Thu Jul 31 14:27:57 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Thu, 31 Jul 2008 14:27:57 +0200 (CEST) Subject: [pypy-svn] r56873 - pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test Message-ID: <20080731122757.5A21A168550@codespeak.net> Author: witulski Date: Thu Jul 31 14:27:57 2008 New Revision: 56873 Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Log: Added a second testline Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Thu Jul 31 14:27:57 2008 @@ -13,4 +13,6 @@ def test_add(): mc = AsmTest() mc.ADD("rax", "r11") - assert mc.get_as_string() == "\x49\x00\xC3" \ No newline at end of file + assert mc.get_as_string() == "\x49\x00\xC3" + mc.ADD("rbx", "rbx") + assert mc.get_as_string() == "\x49\x00\xC3\x48\x00\xDB" \ No newline at end of file From witulski at codespeak.net Thu Jul 31 15:42:40 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Thu, 31 Jul 2008 15:42:40 +0200 (CEST) Subject: [pypy-svn] r56876 - in pypy/branch/oo-jit/pypy/jit/codegen/x86_64: . test Message-ID: <20080731134240.59CE6169E20@codespeak.net> Author: witulski Date: Thu Jul 31 15:42:39 2008 New Revision: 56876 Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Log: (cfBolz, witulski) Changed ADD. created MOV Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py Thu Jul 31 15:42:39 2008 @@ -18,6 +18,18 @@ "r15": (1, 7), } +def make_two_quadreg_instr(opcode): + # XXX for now, arg1 and arg2 are registers + def quadreg_instr(self, arg1, arg2): + rexR, modrm1 = self.get_register_bits(arg1) + rexB, modrm2 = self.get_register_bits(arg2) + #rexW(1) = 64bitMode rexX(0) = doesn't matter + self.write_rex_byte(1, rexR, 0, rexB) + self.write(opcode) + self.write_modRM_byte(3, modrm1, modrm2) + return quadreg_instr + + class X86_64CodeBuilder(object): """ creats x86_64 opcodes""" def write(self, data): @@ -28,15 +40,9 @@ """ tells the current position in memory""" raise NotImplementedError - # XXX for now, arg1 and arg2 are registers - def ADD(self, arg1, arg2): - rexR, modrm1 = self.get_register_bits(arg1) - rexB, modrm2 = self.get_register_bits(arg2) - #rexW(1) = 64bitMode rexX(0) = doesn't matter - self.write_rex_byte(1, rexR, 0, rexB) - self.write('\x00') - self.write_modRM_byte(3, modrm1, modrm2) - + ADD = make_two_quadreg_instr("\x00") + MOV = make_two_quadreg_instr("\x8B") + def get_register_bits(self, register): return REGISTER_MAP[register] @@ -47,4 +53,5 @@ def write_modRM_byte(self, mod, reg, rm): byte = mod << 6 | (reg << 3) | rm - self.write(chr(byte)) \ No newline at end of file + self.write(chr(byte)) + Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Thu Jul 31 15:42:39 2008 @@ -15,4 +15,9 @@ mc.ADD("rax", "r11") assert mc.get_as_string() == "\x49\x00\xC3" mc.ADD("rbx", "rbx") - assert mc.get_as_string() == "\x49\x00\xC3\x48\x00\xDB" \ No newline at end of file + assert mc.get_as_string() == "\x49\x00\xC3\x48\x00\xDB" + +def test_mov(): + mc = AsmTest() + mc.MOV("r15","rsp") + assert mc.get_as_string() == "\x4C\x8B\xFC" \ No newline at end of file From witulski at codespeak.net Thu Jul 31 17:28:14 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Thu, 31 Jul 2008 17:28:14 +0200 (CEST) Subject: [pypy-svn] r56878 - in pypy/branch/oo-jit/pypy/jit/codegen/x86_64: . test Message-ID: <20080731152814.B8FDB169E15@codespeak.net> Author: witulski Date: Thu Jul 31 17:28:13 2008 New Revision: 56878 Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Log: (cfbolz, witulski) Fixed: Argument registers(and rex) swapped Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py Thu Jul 31 17:28:13 2008 @@ -24,9 +24,10 @@ rexR, modrm1 = self.get_register_bits(arg1) rexB, modrm2 = self.get_register_bits(arg2) #rexW(1) = 64bitMode rexX(0) = doesn't matter - self.write_rex_byte(1, rexR, 0, rexB) + # exchange the two arguments (rexB/rexR) (modrm2/modrm1) + self.write_rex_byte(1, rexB, 0, rexR) self.write(opcode) - self.write_modRM_byte(3, modrm1, modrm2) + self.write_modRM_byte(3, modrm2, modrm1) return quadreg_instr @@ -41,7 +42,10 @@ raise NotImplementedError ADD = make_two_quadreg_instr("\x00") - MOV = make_two_quadreg_instr("\x8B") + MOV = make_two_quadreg_instr("\x89") + + def RET(self): + self.write("\xC3") def get_register_bits(self, register): return REGISTER_MAP[register] Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py Thu Jul 31 17:28:13 2008 @@ -13,11 +13,11 @@ def test_add(): mc = AsmTest() mc.ADD("rax", "r11") - assert mc.get_as_string() == "\x49\x00\xC3" + assert mc.get_as_string() == "\x4C\x00\xD8" mc.ADD("rbx", "rbx") - assert mc.get_as_string() == "\x49\x00\xC3\x48\x00\xDB" + assert mc.get_as_string() == "\x4C\x00\xD8\x48\x00\xDB" def test_mov(): mc = AsmTest() mc.MOV("r15","rsp") - assert mc.get_as_string() == "\x4C\x8B\xFC" \ No newline at end of file + assert mc.get_as_string() == "\x49\x89\xE7" \ No newline at end of file From witulski at codespeak.net Thu Jul 31 17:31:49 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Thu, 31 Jul 2008 17:31:49 +0200 (CEST) Subject: [pypy-svn] r56879 - in pypy/branch/oo-jit/pypy/jit/codegen/x86_64: . test Message-ID: <20080731153149.47C38169E7B@codespeak.net> Author: witulski Date: Thu Jul 31 17:31:48 2008 New Revision: 56879 Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/codebuf.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/objmodel.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py Log: (cfbolz, witulski) start the actual jit backend. a very simple example adding two integers works. Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/codebuf.py ============================================================================== --- (empty file) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/codebuf.py Thu Jul 31 17:31:48 2008 @@ -0,0 +1,34 @@ +import os +from pypy.rpython.lltypesystem import lltype, rffi +from pypy.jit.codegen.x86_64.assembler import X86_64CodeBuilder +#from ri386 import I386CodeBuilder + +modname = 'pypy.jit.codegen.i386.codebuf_' + os.name +memhandler = __import__(modname, globals(), locals(), ['__doc__']) + +PTR = memhandler.PTR + +class InMemoryCodeBuilder(X86_64CodeBuilder): + + def __init__(self, map_size): + data = memhandler.alloc(map_size) + self._data = data + self._size = map_size + self._pos = 0 + self._all = [] + + def write(self, data): + p = self._pos + if p + len(data) > self._size: + raise CodeBlockOverflow + for c in data: + self._data[p] = c + p += 1 + self._all.append(data) + self._pos = p + + def tell(self): + baseaddr = rffi.cast(lltype.Signed, self._data) + return baseaddr + self._pos + + Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/objmodel.py ============================================================================== --- (empty file) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/objmodel.py Thu Jul 31 17:31:48 2008 @@ -0,0 +1,10 @@ +from pypy.jit.codegen import model + + +class IntVar(model.GenVar): + def __init__(self, reg): + self.reg = reg + +class Const(model.GenConst): + def __init__(self, value): + self.value = value \ No newline at end of file Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py ============================================================================== --- (empty file) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py Thu Jul 31 17:31:48 2008 @@ -0,0 +1,74 @@ +from pypy.jit.codegen import model +from pypy.rlib.objectmodel import specialize +from pypy.jit.codegen.x86_64.objmodel import IntVar, Const +from pypy.jit.codegen.x86_64.codebuf import InMemoryCodeBuilder + +class Builder(model.GenBuilder): + + MC_SIZE = 65536 + + def __init__(self): + self.mc = InMemoryCodeBuilder(self.MC_SIZE) + self.freeregisters ={ + "rax":None, + "rcx":None, + "rdx":None, + # "rbx":None, + "rsi":None, + "rdi":None, + "r8": None, + "r9": None, + "r10":None, + # "r11":None, + # "r12":None, + # "r13":None, + # "r14":None, + # "r15":None, + } + + @specialize.arg(1) + def genop1(self, opname, gv_arg): + genmethod = getattr(self, 'op_' + opname) + return genmethod(gv_arg) + + @specialize.arg(1) + def genop2(self, opname, gv_arg1, gv_arg2): + genmethod = getattr(self, 'op_' + opname) + return genmethod(gv_arg1, gv_arg2) + + def op_int_add(self, gv_x, gv_y): + gv_z = self.allocate_register() + self.mc.MOV(gv_z.reg, gv_x.reg) + self.mc.ADD(gv_z.reg, gv_y.reg) + return gv_z + + def finish_and_return(self, sigtoken, gv_returnvar): + #self.mc.write("\xB8\x0F\x00\x00\x00") + self.mc.MOV("rax", gv_returnvar.reg) + self.mc.RET() + + def allocate_register(self, register=None): + if register is None: + return IntVar(self.freeregisters.popitem()[0]) + else: + del self.freeregisters[register] + return IntVar(register) + + +class RX86_64GenOp(model.AbstractRGenOp): + + + @staticmethod + @specialize.memo() + def sigToken(FUNCTYPE): + return None + + def newgraph(self, sigtoken, name): + # XXX for now assume that all functions take two ints and return an int + builder = Builder() + inputargs_gv = [builder.allocate_register("rdi"), + builder.allocate_register("rsi")] + #XXX + entrypoint = Const(builder.mc.tell()) + return builder, entrypoint, inputargs_gv + Added: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py ============================================================================== --- (empty file) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py Thu Jul 31 17:31:48 2008 @@ -0,0 +1,32 @@ +from pypy.jit.codegen.x86_64.rgenop import RX86_64GenOp +from pypy.rpython.lltypesystem import lltype +from ctypes import cast, c_void_p, CFUNCTYPE, c_long, c_double + +def make_testbuilder(): + rgenop = RX86_64GenOp() + FUNC = lltype.FuncType([lltype.Signed, lltype.Signed], lltype.Signed) #the funtiontype(arguments,returntype) of the graph we will create + token = rgenop.sigToken(FUNC) + builder, entrypoint, inputargs_gv = rgenop.newgraph(token, "test") + builder.start_writing() + ctypestypes = [c_long, c_long] + fp = cast(c_void_p(entrypoint.value), + CFUNCTYPE(c_long, *ctypestypes)) + return builder, fp, inputargs_gv, token + +def test_add(): + builder, fp, inputargs_gv, token = make_testbuilder() + genv0 = inputargs_gv[0] #the first argument "place" + genv1 = inputargs_gv[1] + genv_result = builder.genop2("int_add", genv0, genv1) #creates the addition and returns the place(register) of the result in genv_result + builder.finish_and_return(token, genv_result) + ten = fp(4, 6) + assert ten == 10 + print ten + +def test_ret(): + builder, fp, inputargs_gv, token = make_testbuilder() + builder.finish_and_return(token, inputargs_gv[0]) + print repr("".join(builder.mc._all)) + four = fp(4, 17) + assert four == 4 + print four \ No newline at end of file From witulski at codespeak.net Thu Jul 31 17:44:22 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Thu, 31 Jul 2008 17:44:22 +0200 (CEST) Subject: [pypy-svn] r56880 - in pypy/branch/oo-jit/pypy/jit/codegen/x86_64: . test Message-ID: <20080731154422.1CD7D169E54@codespeak.net> Author: witulski Date: Thu Jul 31 17:44:21 2008 New Revision: 56880 Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py Log: (witulski) Added SUB instruction Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py Thu Jul 31 17:44:21 2008 @@ -43,6 +43,7 @@ ADD = make_two_quadreg_instr("\x00") MOV = make_two_quadreg_instr("\x89") + SUB = make_two_quadreg_instr("\x28") def RET(self): self.write("\xC3") Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py Thu Jul 31 17:44:21 2008 @@ -42,6 +42,12 @@ self.mc.ADD(gv_z.reg, gv_y.reg) return gv_z + def op_int_sub(self, gv_x, gv_y): + gv_z = self.allocate_register() + self.mc.MOV(gv_z.reg, gv_x.reg) + self.mc.SUB(gv_z.reg, gv_y.reg) + return gv_z + def finish_and_return(self, sigtoken, gv_returnvar): #self.mc.write("\xB8\x0F\x00\x00\x00") self.mc.MOV("rax", gv_returnvar.reg) Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py Thu Jul 31 17:44:21 2008 @@ -29,4 +29,14 @@ print repr("".join(builder.mc._all)) four = fp(4, 17) assert four == 4 + print four + +def test_sub(): + builder, fp, inputargs_gv, token = make_testbuilder() + genv0 = inputargs_gv[0] #the first argument "place" + genv1 = inputargs_gv[1] + genv_result = builder.genop2("int_sub", genv0, genv1) #creates the addition and returns the place(register) of the result in genv_result + builder.finish_and_return(token, genv_result) + four = fp(10, 6) + assert four == 4 print four \ No newline at end of file From cfbolz at codespeak.net Thu Jul 31 17:48:54 2008 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 31 Jul 2008 17:48:54 +0200 (CEST) Subject: [pypy-svn] r56881 - in pypy/branch/oo-jit/pypy/jit/codegen: cli cli/test x86_64 x86_64/test Message-ID: <20080731154854.B48A0169E7C@codespeak.net> Author: cfbolz Date: Thu Jul 31 17:48:52 2008 New Revision: 56881 Modified: pypy/branch/oo-jit/pypy/jit/codegen/cli/ (props changed) pypy/branch/oo-jit/pypy/jit/codegen/cli/test/ (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/ (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/__init__.py (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/assembler.py (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/codebuf.py (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/objmodel.py (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/ (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/__init__.py (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_assembler.py (props changed) pypy/branch/oo-jit/pypy/jit/codegen/x86_64/test/test_simple.py (props changed) Log: FIXEOL From witulski at codespeak.net Thu Jul 31 18:03:27 2008 From: witulski at codespeak.net (witulski at codespeak.net) Date: Thu, 31 Jul 2008 18:03:27 +0200 (CEST) Subject: [pypy-svn] r56882 - pypy/branch/oo-jit/pypy/jit/codegen/x86_64 Message-ID: <20080731160327.3149E169DB5@codespeak.net> Author: witulski Date: Thu Jul 31 18:03:25 2008 New Revision: 56882 Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py Log: (cfbolz,witulski) refactored add and sub Modified: pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py ============================================================================== --- pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py (original) +++ pypy/branch/oo-jit/pypy/jit/codegen/x86_64/rgenop.py Thu Jul 31 18:03:25 2008 @@ -3,12 +3,23 @@ from pypy.jit.codegen.x86_64.objmodel import IntVar, Const from pypy.jit.codegen.x86_64.codebuf import InMemoryCodeBuilder +def make_two_argument_method(name): + def op_int(self, gv_x, gv_y): + gv_z = self.allocate_register() + self.mc.MOV(gv_z.reg, gv_x.reg) + method = getattr(self.mc, name) + method(gv_z.reg, gv_y.reg) + return gv_z + return op_int + + class Builder(model.GenBuilder): MC_SIZE = 65536 def __init__(self): self.mc = InMemoryCodeBuilder(self.MC_SIZE) + #callee-saved registers are commented out self.freeregisters ={ "rax":None, "rcx":None, @@ -36,17 +47,8 @@ genmethod = getattr(self, 'op_' + opname) return genmethod(gv_arg1, gv_arg2) - def op_int_add(self, gv_x, gv_y): - gv_z = self.allocate_register() - self.mc.MOV(gv_z.reg, gv_x.reg) - self.mc.ADD(gv_z.reg, gv_y.reg) - return gv_z - - def op_int_sub(self, gv_x, gv_y): - gv_z = self.allocate_register() - self.mc.MOV(gv_z.reg, gv_x.reg) - self.mc.SUB(gv_z.reg, gv_y.reg) - return gv_z + op_int_add = make_two_argument_method("ADD") + op_int_sub = make_two_argument_method("SUB") def finish_and_return(self, sigtoken, gv_returnvar): #self.mc.write("\xB8\x0F\x00\x00\x00")